2023-04-28 04:55:27,005 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6 2023-04-28 04:55:27,018 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-28 04:55:27,032 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-28 04:55:27,032 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/cluster_ed675f99-8f25-1dee-0ca7-08f146b620e3, deleteOnExit=true 2023-04-28 04:55:27,032 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-28 04:55:27,033 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/test.cache.data in system properties and HBase conf 2023-04-28 04:55:27,033 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/hadoop.tmp.dir in system properties and HBase conf 2023-04-28 04:55:27,033 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/hadoop.log.dir in system properties and HBase conf 2023-04-28 04:55:27,034 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-28 04:55:27,035 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-28 04:55:27,035 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-28 04:55:27,123 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-28 04:55:27,476 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-28 04:55:27,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-28 04:55:27,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-28 04:55:27,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-28 04:55:27,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-28 04:55:27,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-28 04:55:27,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-28 04:55:27,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-28 04:55:27,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-28 04:55:27,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-28 04:55:27,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/nfs.dump.dir in system properties and HBase conf 2023-04-28 04:55:27,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/java.io.tmpdir in system properties and HBase conf 2023-04-28 04:55:27,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-28 04:55:27,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-28 04:55:27,484 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-28 04:55:27,918 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-28 04:55:27,922 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-28 04:55:28,688 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-28 04:55:28,842 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-28 04:55:28,860 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-28 04:55:28,892 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-28 04:55:28,924 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/java.io.tmpdir/Jetty_localhost_localdomain_45073_hdfs____ffnzk5/webapp 2023-04-28 04:55:29,027 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:45073 2023-04-28 04:55:29,054 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-28 04:55:29,055 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-28 04:55:29,587 WARN [Listener at localhost.localdomain/42461] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-28 04:55:29,698 WARN [Listener at localhost.localdomain/42461] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-28 04:55:29,720 WARN [Listener at localhost.localdomain/42461] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-28 04:55:29,728 INFO [Listener at localhost.localdomain/42461] log.Slf4jLog(67): jetty-6.1.26 2023-04-28 04:55:29,733 INFO [Listener at localhost.localdomain/42461] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/java.io.tmpdir/Jetty_localhost_43373_datanode____.ah7grt/webapp 2023-04-28 04:55:29,814 INFO [Listener at localhost.localdomain/42461] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43373 2023-04-28 04:55:30,150 WARN [Listener at localhost.localdomain/36983] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-28 04:55:31,242 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd02891933e1849ed: Processing first storage report for DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409 from datanode 8cc9c9b5-5dd2-430f-8a4b-623291ea4ba7 2023-04-28 04:55:31,244 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd02891933e1849ed: from storage DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409 node DatanodeRegistration(127.0.0.1:32811, datanodeUuid=8cc9c9b5-5dd2-430f-8a4b-623291ea4ba7, infoPort=46539, infoSecurePort=0, ipcPort=36983, storageInfo=lv=-57;cid=testClusterID;nsid=420633454;c=1682657727979), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-28 04:55:31,244 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd02891933e1849ed: Processing first storage report for DS-583f41c0-6358-42ba-abbf-e47d9f5f9ee1 from datanode 8cc9c9b5-5dd2-430f-8a4b-623291ea4ba7 2023-04-28 04:55:31,244 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd02891933e1849ed: from storage DS-583f41c0-6358-42ba-abbf-e47d9f5f9ee1 node DatanodeRegistration(127.0.0.1:32811, datanodeUuid=8cc9c9b5-5dd2-430f-8a4b-623291ea4ba7, infoPort=46539, infoSecurePort=0, ipcPort=36983, storageInfo=lv=-57;cid=testClusterID;nsid=420633454;c=1682657727979), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-28 04:55:31,263 DEBUG [Listener at localhost.localdomain/36983] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6 2023-04-28 04:55:31,311 INFO [Listener at localhost.localdomain/36983] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/cluster_ed675f99-8f25-1dee-0ca7-08f146b620e3/zookeeper_0, clientPort=52932, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/cluster_ed675f99-8f25-1dee-0ca7-08f146b620e3/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/cluster_ed675f99-8f25-1dee-0ca7-08f146b620e3/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-28 04:55:31,323 INFO [Listener at localhost.localdomain/36983] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52932 2023-04-28 04:55:31,332 INFO [Listener at localhost.localdomain/36983] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:31,335 INFO [Listener at localhost.localdomain/36983] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:31,991 INFO [Listener at localhost.localdomain/36983] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c with version=8 2023-04-28 04:55:31,991 INFO [Listener at localhost.localdomain/36983] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/hbase-staging 2023-04-28 04:55:32,264 INFO [Listener at localhost.localdomain/36983] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-28 04:55:32,633 INFO [Listener at localhost.localdomain/36983] client.ConnectionUtils(127): master/jenkins-hbase12:0 server-side Connection retries=45 2023-04-28 04:55:32,673 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 04:55:32,673 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-28 04:55:32,674 INFO [Listener at localhost.localdomain/36983] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-28 04:55:32,674 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 04:55:32,674 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-28 04:55:32,816 INFO [Listener at localhost.localdomain/36983] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-28 04:55:32,878 DEBUG [Listener at localhost.localdomain/36983] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-28 04:55:32,963 INFO [Listener at localhost.localdomain/36983] ipc.NettyRpcServer(120): Bind to /136.243.104.168:38117 2023-04-28 04:55:32,972 INFO [Listener at localhost.localdomain/36983] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:32,974 INFO [Listener at localhost.localdomain/36983] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:32,990 INFO [Listener at localhost.localdomain/36983] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38117 connecting to ZooKeeper ensemble=127.0.0.1:52932 2023-04-28 04:55:33,138 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:381170x0, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-28 04:55:33,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38117-0x100f60986970000 connected 2023-04-28 04:55:33,232 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 04:55:33,233 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 04:55:33,238 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-28 04:55:33,249 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38117 2023-04-28 04:55:33,249 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38117 2023-04-28 04:55:33,249 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38117 2023-04-28 04:55:33,250 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38117 2023-04-28 04:55:33,250 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38117 2023-04-28 04:55:33,255 INFO [Listener at localhost.localdomain/36983] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c, hbase.cluster.distributed=false 2023-04-28 04:55:33,314 INFO [Listener at localhost.localdomain/36983] client.ConnectionUtils(127): regionserver/jenkins-hbase12:0 server-side Connection retries=45 2023-04-28 04:55:33,314 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 04:55:33,314 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-28 04:55:33,314 INFO [Listener at localhost.localdomain/36983] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-28 04:55:33,315 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 04:55:33,315 INFO [Listener at localhost.localdomain/36983] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-28 04:55:33,319 INFO [Listener at localhost.localdomain/36983] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-28 04:55:33,322 INFO [Listener at localhost.localdomain/36983] ipc.NettyRpcServer(120): Bind to /136.243.104.168:37663 2023-04-28 04:55:33,325 INFO [Listener at localhost.localdomain/36983] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-28 04:55:33,329 DEBUG [Listener at localhost.localdomain/36983] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-28 04:55:33,330 INFO [Listener at localhost.localdomain/36983] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:33,333 INFO [Listener at localhost.localdomain/36983] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:33,334 INFO [Listener at localhost.localdomain/36983] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37663 connecting to ZooKeeper ensemble=127.0.0.1:52932 2023-04-28 04:55:33,436 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:376630x0, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-28 04:55:33,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37663-0x100f60986970001 connected 2023-04-28 04:55:33,437 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 04:55:33,438 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 04:55:33,439 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-28 04:55:33,440 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37663 2023-04-28 04:55:33,440 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37663 2023-04-28 04:55:33,440 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37663 2023-04-28 04:55:33,444 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37663 2023-04-28 04:55:33,444 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37663 2023-04-28 04:55:33,446 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:33,629 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-28 04:55:33,630 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:33,660 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-28 04:55:33,660 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-28 04:55:33,660 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:33,661 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-28 04:55:33,662 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase12.apache.org,38117,1682657732105 from backup master directory 2023-04-28 04:55:33,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-28 04:55:33,670 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:33,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-28 04:55:33,671 WARN [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-28 04:55:33,671 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:33,674 INFO [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-28 04:55:33,675 INFO [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-28 04:55:33,751 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/hbase.id with ID: fad9d559-b7d3-484e-ade8-07b94f8a1a0d 2023-04-28 04:55:33,794 INFO [master/jenkins-hbase12:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 04:55:33,818 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:33,857 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4cf0e276 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:33,895 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e132f72, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:33,917 INFO [master/jenkins-hbase12:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:55:33,920 INFO [master/jenkins-hbase12:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-28 04:55:33,935 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-28 04:55:33,936 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-28 04:55:33,938 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:33,941 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:33,942 INFO [master/jenkins-hbase12:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 04:55:33,971 INFO [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store-tmp 2023-04-28 04:55:33,997 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:33,997 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-28 04:55:33,997 INFO [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:55:33,997 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:55:33,997 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-28 04:55:33,997 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:55:33,998 INFO [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:55:33,998 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 04:55:34,000 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/WALs/jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:34,020 INFO [master/jenkins-hbase12:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase12.apache.org%2C38117%2C1682657732105, suffix=, logDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/WALs/jenkins-hbase12.apache.org,38117,1682657732105, archiveDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/oldWALs, maxLogs=10 2023-04-28 04:55:34,074 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32811,DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409,DISK] 2023-04-28 04:55:34,081 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:34,134 INFO [master/jenkins-hbase12:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/WALs/jenkins-hbase12.apache.org,38117,1682657732105/jenkins-hbase12.apache.org%2C38117%2C1682657732105.1682657734030 2023-04-28 04:55:34,134 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32811,DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409,DISK]] 2023-04-28 04:55:34,135 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:55:34,135 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:34,138 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 04:55:34,139 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 04:55:34,193 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-28 04:55:34,199 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-28 04:55:34,225 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-28 04:55:34,239 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:34,246 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-28 04:55:34,247 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-28 04:55:34,264 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 04:55:34,268 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:55:34,270 INFO [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68918012, jitterRate=0.0269584059715271}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-28 04:55:34,270 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 04:55:34,272 INFO [master/jenkins-hbase12:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-28 04:55:34,295 INFO [master/jenkins-hbase12:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-28 04:55:34,295 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-28 04:55:34,298 INFO [master/jenkins-hbase12:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-28 04:55:34,300 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-28 04:55:34,336 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 36 msec 2023-04-28 04:55:34,337 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-28 04:55:34,362 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-28 04:55:34,369 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-28 04:55:34,402 INFO [master/jenkins-hbase12:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-28 04:55:34,407 INFO [master/jenkins-hbase12:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-28 04:55:34,409 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-28 04:55:34,415 INFO [master/jenkins-hbase12:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-28 04:55:34,421 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-28 04:55:34,439 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:34,440 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-28 04:55:34,441 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-28 04:55:34,456 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-28 04:55:34,470 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-28 04:55:34,470 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-28 04:55:34,470 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:34,471 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase12.apache.org,38117,1682657732105, sessionid=0x100f60986970000, setting cluster-up flag (Was=false) 2023-04-28 04:55:34,502 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:34,534 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-28 04:55:34,535 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:34,554 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:34,586 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-28 04:55:34,588 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:34,590 WARN [master/jenkins-hbase12:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.hbase-snapshot/.tmp 2023-04-28 04:55:34,649 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(952): ClusterId : fad9d559-b7d3-484e-ade8-07b94f8a1a0d 2023-04-28 04:55:34,654 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-28 04:55:34,671 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-28 04:55:34,671 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-28 04:55:34,682 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-28 04:55:34,683 DEBUG [RS:0;jenkins-hbase12:37663] zookeeper.ReadOnlyZKClient(139): Connect 0x02dca4c8 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:34,703 DEBUG [RS:0;jenkins-hbase12:37663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c1b4ae3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:34,704 DEBUG [RS:0;jenkins-hbase12:37663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@824cfb8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase12.apache.org/136.243.104.168:0 2023-04-28 04:55:34,705 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-28 04:55:34,713 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase12:0, corePoolSize=5, maxPoolSize=5 2023-04-28 04:55:34,713 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase12:0, corePoolSize=5, maxPoolSize=5 2023-04-28 04:55:34,713 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase12:0, corePoolSize=5, maxPoolSize=5 2023-04-28 04:55:34,713 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase12:0, corePoolSize=5, maxPoolSize=5 2023-04-28 04:55:34,713 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase12:0, corePoolSize=10, maxPoolSize=10 2023-04-28 04:55:34,714 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,714 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase12:0, corePoolSize=2, maxPoolSize=2 2023-04-28 04:55:34,714 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,715 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682657764715 2023-04-28 04:55:34,716 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-28 04:55:34,720 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-28 04:55:34,721 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-28 04:55:34,725 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-28 04:55:34,727 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-28 04:55:34,733 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-28 04:55:34,733 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-28 04:55:34,733 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-28 04:55:34,734 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-28 04:55:34,734 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase12:37663 2023-04-28 04:55:34,735 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,736 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-28 04:55:34,737 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-28 04:55:34,738 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-28 04:55:34,739 INFO [RS:0;jenkins-hbase12:37663] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-28 04:55:34,739 INFO [RS:0;jenkins-hbase12:37663] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-28 04:55:34,739 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1023): About to register with Master. 2023-04-28 04:55:34,740 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-28 04:55:34,741 INFO [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-28 04:55:34,742 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase12:0:becomeActiveMaster-HFileCleaner.large.0-1682657734742,5,FailOnTimeoutGroup] 2023-04-28 04:55:34,743 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase12.apache.org,38117,1682657732105 with isa=jenkins-hbase12.apache.org/136.243.104.168:37663, startcode=1682657733313 2023-04-28 04:55:34,744 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase12:0:becomeActiveMaster-HFileCleaner.small.0-1682657734743,5,FailOnTimeoutGroup] 2023-04-28 04:55:34,744 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,745 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-28 04:55:34,748 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,748 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,764 DEBUG [RS:0;jenkins-hbase12:37663] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-28 04:55:34,807 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:54463, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-28 04:55:34,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.ServerManager(394): Registering regionserver=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:34,830 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c 2023-04-28 04:55:34,831 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:42461 2023-04-28 04:55:34,831 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-28 04:55:34,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 04:55:34,848 DEBUG [RS:0;jenkins-hbase12:37663] zookeeper.ZKUtil(162): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:34,848 WARN [RS:0;jenkins-hbase12:37663] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-28 04:55:34,849 INFO [RS:0;jenkins-hbase12:37663] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 04:55:34,849 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/WALs/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:34,851 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase12.apache.org,37663,1682657733313] 2023-04-28 04:55:34,858 DEBUG [RS:0;jenkins-hbase12:37663] zookeeper.ZKUtil(162): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:34,868 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-28 04:55:34,875 INFO [RS:0;jenkins-hbase12:37663] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-28 04:55:34,896 INFO [RS:0;jenkins-hbase12:37663] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-28 04:55:34,901 INFO [RS:0;jenkins-hbase12:37663] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-28 04:55:34,901 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,902 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-28 04:55:34,911 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,911 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,911 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,912 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,912 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,912 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,912 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase12:0, corePoolSize=2, maxPoolSize=2 2023-04-28 04:55:34,913 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,913 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,913 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,913 DEBUG [RS:0;jenkins-hbase12:37663] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase12:0, corePoolSize=1, maxPoolSize=1 2023-04-28 04:55:34,920 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,920 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,920 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,939 INFO [RS:0;jenkins-hbase12:37663] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-28 04:55:34,941 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase12.apache.org,37663,1682657733313-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:34,960 INFO [RS:0;jenkins-hbase12:37663] regionserver.Replication(203): jenkins-hbase12.apache.org,37663,1682657733313 started 2023-04-28 04:55:34,960 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1638): Serving as jenkins-hbase12.apache.org,37663,1682657733313, RpcServer on jenkins-hbase12.apache.org/136.243.104.168:37663, sessionid=0x100f60986970001 2023-04-28 04:55:34,961 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-28 04:55:34,961 DEBUG [RS:0;jenkins-hbase12:37663] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:34,961 DEBUG [RS:0;jenkins-hbase12:37663] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase12.apache.org,37663,1682657733313' 2023-04-28 04:55:34,961 DEBUG [RS:0;jenkins-hbase12:37663] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:34,962 DEBUG [RS:0;jenkins-hbase12:37663] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:34,962 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-28 04:55:34,962 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-28 04:55:34,962 DEBUG [RS:0;jenkins-hbase12:37663] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:34,962 DEBUG [RS:0;jenkins-hbase12:37663] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase12.apache.org,37663,1682657733313' 2023-04-28 04:55:34,962 DEBUG [RS:0;jenkins-hbase12:37663] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-28 04:55:34,963 DEBUG [RS:0;jenkins-hbase12:37663] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-28 04:55:34,963 DEBUG [RS:0;jenkins-hbase12:37663] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-28 04:55:34,963 INFO [RS:0;jenkins-hbase12:37663] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-28 04:55:34,963 INFO [RS:0;jenkins-hbase12:37663] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-28 04:55:35,077 INFO [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase12.apache.org%2C37663%2C1682657733313, suffix=, logDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/WALs/jenkins-hbase12.apache.org,37663,1682657733313, archiveDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/oldWALs, maxLogs=32 2023-04-28 04:55:35,094 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32811,DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409,DISK] 2023-04-28 04:55:35,098 INFO [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/WALs/jenkins-hbase12.apache.org,37663,1682657733313/jenkins-hbase12.apache.org%2C37663%2C1682657733313.1682657735079 2023-04-28 04:55:35,098 DEBUG [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32811,DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409,DISK]] 2023-04-28 04:55:35,154 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-28 04:55:35,155 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-28 04:55:35,155 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c 2023-04-28 04:55:35,569 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:35,572 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-28 04:55:35,575 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/info 2023-04-28 04:55:35,576 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-28 04:55:35,577 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:35,577 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-28 04:55:35,580 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/rep_barrier 2023-04-28 04:55:35,581 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-28 04:55:35,582 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:35,582 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-28 04:55:35,584 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/table 2023-04-28 04:55:35,585 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-28 04:55:35,586 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:35,588 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740 2023-04-28 04:55:35,589 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740 2023-04-28 04:55:35,592 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:55:35,594 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-28 04:55:35,598 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:55:35,599 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64296152, jitterRate=-0.04191267490386963}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:55:35,599 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-28 04:55:35,599 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-28 04:55:35,599 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-28 04:55:35,599 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-28 04:55:35,599 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-28 04:55:35,599 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-28 04:55:35,600 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-28 04:55:35,600 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-28 04:55:35,605 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-28 04:55:35,605 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-28 04:55:35,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-28 04:55:35,624 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-28 04:55:35,626 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-28 04:55:35,778 DEBUG [jenkins-hbase12:38117] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-28 04:55:35,781 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase12.apache.org,37663,1682657733313, state=OPENING 2023-04-28 04:55:35,807 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-28 04:55:35,817 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:35,818 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-28 04:55:35,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:55:36,021 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:36,023 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-28 04:55:36,027 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-28 04:55:36,039 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-28 04:55:36,039 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 04:55:36,043 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase12.apache.org%2C37663%2C1682657733313.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/WALs/jenkins-hbase12.apache.org,37663,1682657733313, archiveDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/oldWALs, maxLogs=32 2023-04-28 04:55:36,059 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32811,DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409,DISK] 2023-04-28 04:55:36,064 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/WALs/jenkins-hbase12.apache.org,37663,1682657733313/jenkins-hbase12.apache.org%2C37663%2C1682657733313.meta.1682657736044.meta 2023-04-28 04:55:36,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32811,DS-38be9ef7-0f4f-409b-9e1e-1df1a557c409,DISK]] 2023-04-28 04:55:36,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:55:36,068 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-28 04:55:36,081 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-28 04:55:36,085 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-28 04:55:36,089 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-28 04:55:36,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:36,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-28 04:55:36,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-28 04:55:36,092 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-28 04:55:36,094 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/info 2023-04-28 04:55:36,094 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/info 2023-04-28 04:55:36,094 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-28 04:55:36,095 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:36,095 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-28 04:55:36,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/rep_barrier 2023-04-28 04:55:36,097 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/rep_barrier 2023-04-28 04:55:36,097 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-28 04:55:36,098 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:36,098 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-28 04:55:36,099 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/table 2023-04-28 04:55:36,099 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/table 2023-04-28 04:55:36,100 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-28 04:55:36,101 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:36,102 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740 2023-04-28 04:55:36,105 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740 2023-04-28 04:55:36,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:55:36,110 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-28 04:55:36,111 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63064686, jitterRate=-0.06026294827461243}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:55:36,111 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-28 04:55:36,119 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682657736014 2023-04-28 04:55:36,134 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-28 04:55:36,134 INFO [RS_OPEN_META-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-28 04:55:36,137 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase12.apache.org,37663,1682657733313, state=OPEN 2023-04-28 04:55:36,192 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-28 04:55:36,193 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-28 04:55:36,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-28 04:55:36,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase12.apache.org,37663,1682657733313 in 372 msec 2023-04-28 04:55:36,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-28 04:55:36,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 589 msec 2023-04-28 04:55:36,212 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5740 sec 2023-04-28 04:55:36,212 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682657736212, completionTime=-1 2023-04-28 04:55:36,212 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-28 04:55:36,213 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-28 04:55:36,265 DEBUG [hconnection-0x76732ff5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:36,268 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:36,285 INFO [master/jenkins-hbase12:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-28 04:55:36,285 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682657796285 2023-04-28 04:55:36,285 INFO [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682657856285 2023-04-28 04:55:36,285 INFO [master/jenkins-hbase12:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 72 msec 2023-04-28 04:55:36,377 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase12.apache.org,38117,1682657732105-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:36,378 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase12.apache.org,38117,1682657732105-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:36,378 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase12.apache.org,38117,1682657732105-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:36,379 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase12:38117, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:36,379 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-28 04:55:36,385 DEBUG [master/jenkins-hbase12:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-28 04:55:36,391 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-28 04:55:36,392 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-28 04:55:36,400 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-28 04:55:36,403 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:55:36,406 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:55:36,429 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,431 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae empty. 2023-04-28 04:55:36,432 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,432 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-28 04:55:36,468 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-28 04:55:36,470 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => b7a9f863068c5f23d85b7830bfb4ddae, NAME => 'hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:55:36,484 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:36,484 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing b7a9f863068c5f23d85b7830bfb4ddae, disabling compactions & flushes 2023-04-28 04:55:36,484 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,484 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,484 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. after waiting 0 ms 2023-04-28 04:55:36,484 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,484 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,484 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for b7a9f863068c5f23d85b7830bfb4ddae: 2023-04-28 04:55:36,490 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:55:36,503 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682657736492"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657736492"}]},"ts":"1682657736492"} 2023-04-28 04:55:36,528 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:55:36,530 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:55:36,534 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657736530"}]},"ts":"1682657736530"} 2023-04-28 04:55:36,540 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-28 04:55:36,608 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b7a9f863068c5f23d85b7830bfb4ddae, ASSIGN}] 2023-04-28 04:55:36,611 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b7a9f863068c5f23d85b7830bfb4ddae, ASSIGN 2023-04-28 04:55:36,614 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b7a9f863068c5f23d85b7830bfb4ddae, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:55:36,765 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b7a9f863068c5f23d85b7830bfb4ddae, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:36,765 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682657736765"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657736765"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657736765"}]},"ts":"1682657736765"} 2023-04-28 04:55:36,770 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure b7a9f863068c5f23d85b7830bfb4ddae, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:55:36,937 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,938 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b7a9f863068c5f23d85b7830bfb4ddae, NAME => 'hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:55:36,939 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,939 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:36,939 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,939 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,942 INFO [StoreOpener-b7a9f863068c5f23d85b7830bfb4ddae-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,944 DEBUG [StoreOpener-b7a9f863068c5f23d85b7830bfb4ddae-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/info 2023-04-28 04:55:36,944 DEBUG [StoreOpener-b7a9f863068c5f23d85b7830bfb4ddae-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/info 2023-04-28 04:55:36,945 INFO [StoreOpener-b7a9f863068c5f23d85b7830bfb4ddae-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7a9f863068c5f23d85b7830bfb4ddae columnFamilyName info 2023-04-28 04:55:36,946 INFO [StoreOpener-b7a9f863068c5f23d85b7830bfb4ddae-1] regionserver.HStore(310): Store=b7a9f863068c5f23d85b7830bfb4ddae/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:36,948 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,949 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,954 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:55:36,964 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:55:36,965 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened b7a9f863068c5f23d85b7830bfb4ddae; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64806950, jitterRate=-0.034301191568374634}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-28 04:55:36,965 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for b7a9f863068c5f23d85b7830bfb4ddae: 2023-04-28 04:55:36,967 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae., pid=6, masterSystemTime=1682657736929 2023-04-28 04:55:36,973 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,973 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:55:36,973 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b7a9f863068c5f23d85b7830bfb4ddae, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:36,974 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682657736972"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657736972"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657736972"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657736972"}]},"ts":"1682657736972"} 2023-04-28 04:55:36,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-28 04:55:36,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure b7a9f863068c5f23d85b7830bfb4ddae, server=jenkins-hbase12.apache.org,37663,1682657733313 in 210 msec 2023-04-28 04:55:36,995 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-28 04:55:36,996 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b7a9f863068c5f23d85b7830bfb4ddae, ASSIGN in 378 msec 2023-04-28 04:55:36,997 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:55:36,998 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657736998"}]},"ts":"1682657736998"} 2023-04-28 04:55:37,001 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-28 04:55:37,051 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-28 04:55:37,051 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:55:37,054 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 658 msec 2023-04-28 04:55:37,059 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-28 04:55:37,060 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:55:37,091 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-28 04:55:37,112 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-28 04:55:37,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 42 msec 2023-04-28 04:55:37,137 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-28 04:55:37,154 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-28 04:55:37,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 32 msec 2023-04-28 04:55:37,196 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-28 04:55:37,217 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-28 04:55:37,218 INFO [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.546sec 2023-04-28 04:55:37,220 INFO [master/jenkins-hbase12:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-28 04:55:37,221 INFO [master/jenkins-hbase12:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-28 04:55:37,221 INFO [master/jenkins-hbase12:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-28 04:55:37,223 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase12.apache.org,38117,1682657732105-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-28 04:55:37,223 INFO [master/jenkins-hbase12:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase12.apache.org,38117,1682657732105-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-28 04:55:37,235 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-28 04:55:37,254 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x1c8a8f6d to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:37,270 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3526833, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:37,283 DEBUG [hconnection-0x2ef6ce1d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:37,298 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:37,306 INFO [Listener at localhost.localdomain/36983] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:55:37,323 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=211, ProcessCount=171, AvailableMemoryMB=10913 2023-04-28 04:55:37,339 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:55:37,343 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:39320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:55:37,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:55:37,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$4(2112): Client=jenkins//136.243.104.168 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:55:37,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 04:55:37,358 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:55:37,360 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:55:37,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(697): Client=jenkins//136.243.104.168 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-28 04:55:37,363 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:37,364 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 empty. 2023-04-28 04:55:37,366 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:37,366 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:55:37,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 04:55:37,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 04:55:37,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 04:55:37,784 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:55:37,788 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 32a2cc8961b71f647281a29aaf4ac2d5, NAME => 'TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:55:37,801 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:37,801 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 32a2cc8961b71f647281a29aaf4ac2d5, disabling compactions & flushes 2023-04-28 04:55:37,801 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:37,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:37,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. after waiting 0 ms 2023-04-28 04:55:37,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:37,802 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:37,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:37,807 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:55:37,810 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657737809"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657737809"}]},"ts":"1682657737809"} 2023-04-28 04:55:37,813 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:55:37,815 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:55:37,815 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657737815"}]},"ts":"1682657737815"} 2023-04-28 04:55:37,817 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 04:55:37,976 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, ASSIGN}] 2023-04-28 04:55:37,982 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, ASSIGN 2023-04-28 04:55:37,984 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:55:37,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 04:55:38,136 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=32a2cc8961b71f647281a29aaf4ac2d5, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,136 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657738136"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657738136"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657738136"}]},"ts":"1682657738136"} 2023-04-28 04:55:38,139 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:55:38,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:38,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 32a2cc8961b71f647281a29aaf4ac2d5, NAME => 'TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:55:38,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:55:38,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,302 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,304 DEBUG [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A 2023-04-28 04:55:38,304 DEBUG [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A 2023-04-28 04:55:38,305 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:55:38,306 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 32a2cc8961b71f647281a29aaf4ac2d5 columnFamilyName A 2023-04-28 04:55:38,307 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.HStore(310): Store=32a2cc8961b71f647281a29aaf4ac2d5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:38,307 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,309 DEBUG [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B 2023-04-28 04:55:38,309 DEBUG [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B 2023-04-28 04:55:38,310 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:55:38,310 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 32a2cc8961b71f647281a29aaf4ac2d5 columnFamilyName B 2023-04-28 04:55:38,311 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.HStore(310): Store=32a2cc8961b71f647281a29aaf4ac2d5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:38,311 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,313 DEBUG [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C 2023-04-28 04:55:38,314 DEBUG [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C 2023-04-28 04:55:38,314 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:55:38,314 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 32a2cc8961b71f647281a29aaf4ac2d5 columnFamilyName C 2023-04-28 04:55:38,315 INFO [StoreOpener-32a2cc8961b71f647281a29aaf4ac2d5-1] regionserver.HStore(310): Store=32a2cc8961b71f647281a29aaf4ac2d5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:55:38,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:38,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:55:38,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:55:38,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 32a2cc8961b71f647281a29aaf4ac2d5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70332910, jitterRate=0.04804202914237976}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:55:38,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:38,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., pid=11, masterSystemTime=1682657738292 2023-04-28 04:55:38,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:38,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:38,332 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=32a2cc8961b71f647281a29aaf4ac2d5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,332 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657738332"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657738332"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657738332"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657738332"}]},"ts":"1682657738332"} 2023-04-28 04:55:38,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-28 04:55:38,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 in 196 msec 2023-04-28 04:55:38,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-28 04:55:38,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, ASSIGN in 362 msec 2023-04-28 04:55:38,343 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:55:38,344 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657738343"}]},"ts":"1682657738343"} 2023-04-28 04:55:38,346 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 04:55:38,376 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:55:38,379 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.0220 sec 2023-04-28 04:55:38,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 04:55:38,490 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-28 04:55:38,497 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x2337cbeb to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,534 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3336dbdc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,537 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,541 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43578, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,544 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:55:38,547 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:39330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:55:38,554 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x13e837e0 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,566 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45864842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,566 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x6362e5a0 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,576 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4654132d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,577 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x6be8c20b to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,587 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ca212c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,588 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x046b5e40 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,597 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ecabb06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,598 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbd52be to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,629 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1439cba9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,630 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x38ebb9f9 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,639 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e044e22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,639 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x13112432 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,670 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76d073d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,671 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x6dcc1cc1 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,681 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18f7ec13, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,682 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x7bcd59bb to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:55:38,692 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76ea7817, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:55:38,702 DEBUG [hconnection-0x1ba37edc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,702 DEBUG [hconnection-0x2024bf8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,703 DEBUG [hconnection-0x173f87db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,703 DEBUG [hconnection-0x6a2dbb62-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,703 DEBUG [hconnection-0x19ac861-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,709 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,709 DEBUG [hconnection-0x4084d829-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,709 DEBUG [hconnection-0x2df32c65-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,710 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43612, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,709 DEBUG [hconnection-0x64b06efe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,712 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,716 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,718 DEBUG [hconnection-0xc522e35-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:55:38,718 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:38,724 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,724 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,726 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,729 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:43676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:55:38,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:38,741 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:38,741 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:38,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:38,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:38,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:38,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:38,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:38,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:38,755 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,755 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,755 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:38,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:38,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:38,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:38,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:38,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:38,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:38,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:38,776 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:38,776 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:38,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:38,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:38,784 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 04:55:38,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:38,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:38,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:38,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:38,805 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:38,805 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:38,805 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:38,805 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:38,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:38,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657798878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:38,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657798879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:38,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657798899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:38,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657798908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:38,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657798908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:38,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657738850, max timestamp=1682657738877Num uniques 859; 2023-04-28 04:55:38,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 04:55:38,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657738850, max timestamp=1682657738886Num uniques 859; 2023-04-28 04:55:38,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 04:55:38,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657738850, max timestamp=1682657738878Num uniques 859; 2023-04-28 04:55:38,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 04:55:39,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657799070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657799073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657799075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657799085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657799082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657799190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657799201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657799201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657799201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657799206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,370 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/95f4783a63e6409ca27d1e57969903ed 2023-04-28 04:55:39,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657799407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657799410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657799413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657799413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657799414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,510 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/d50c18c6024a4d0e8735f206e42092f1 2023-04-28 04:55:39,574 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/fb78e05fc31044448358d5fb9ad73e49 2023-04-28 04:55:39,590 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/95f4783a63e6409ca27d1e57969903ed as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/95f4783a63e6409ca27d1e57969903ed 2023-04-28 04:55:39,602 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/95f4783a63e6409ca27d1e57969903ed, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 04:55:39,605 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/d50c18c6024a4d0e8735f206e42092f1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/d50c18c6024a4d0e8735f206e42092f1 2023-04-28 04:55:39,625 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/d50c18c6024a4d0e8735f206e42092f1, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 04:55:39,628 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/fb78e05fc31044448358d5fb9ad73e49 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/fb78e05fc31044448358d5fb9ad73e49 2023-04-28 04:55:39,642 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/fb78e05fc31044448358d5fb9ad73e49, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 04:55:39,646 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=132.99 KB/136185 for 32a2cc8961b71f647281a29aaf4ac2d5 in 863ms, sequenceid=11, compaction requested=false 2023-04-28 04:55:39,648 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 04:55:39,651 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:39,651 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:39,652 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:39,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:39,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:39,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:39,712 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,712 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:39,713 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,713 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:39,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:39,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:39,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29981216[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:39,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:39,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:39,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:39,723 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,723 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:39,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=139.70 KB heapSize=402.87 KB 2023-04-28 04:55:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:39,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657739069, max timestamp=1682657739724Num uniques 1009; 2023-04-28 04:55:39,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-28 04:55:39,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,775 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:39,776 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:39,776 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:39,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:39,780 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:39,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:39,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:39,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:39,780 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:39,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:39,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:55:39,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:39,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:39,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:39,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:55:39,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:39,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,826 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,828 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:39,859 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:39,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:39,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:39,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:39,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:39,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:39,862 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:39,862 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:39,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:39,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:39,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657799858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657799861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657799901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/9f15d1e8d382462182670d60bfc0b6f7 2023-04-28 04:55:39,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657799906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:39,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657799906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:39,962 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:39,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:40,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:40,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657800008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657800008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/ad3ec3b951f1421f8d52ac09c68ba3b4 2023-04-28 04:55:40,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657800009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:40,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:40,025 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:40,025 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,028 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:40,028 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:40,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,038 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,039 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,039 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:40,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:40,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:40,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:40,049 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:40,049 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,053 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:40,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:40,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:40,054 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,054 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:40,054 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:40,054 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:40,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:40,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:40,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:40,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657800056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657800057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:40,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@208923be[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:40,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:40,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/4b5d7860dad24289ad54910ab3327cb6 2023-04-28 04:55:40,091 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:40,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:40,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:40,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:40,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,106 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/9f15d1e8d382462182670d60bfc0b6f7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9f15d1e8d382462182670d60bfc0b6f7 2023-04-28 04:55:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:40,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:40,116 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:40,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:40,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:40,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:40,117 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:40,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9f15d1e8d382462182670d60bfc0b6f7, entries=250, sequenceid=50, filesize=16.2 K 2023-04-28 04:55:40,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/ad3ec3b951f1421f8d52ac09c68ba3b4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ad3ec3b951f1421f8d52ac09c68ba3b4 2023-04-28 04:55:40,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ad3ec3b951f1421f8d52ac09c68ba3b4, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 04:55:40,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/4b5d7860dad24289ad54910ab3327cb6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/4b5d7860dad24289ad54910ab3327cb6 2023-04-28 04:55:40,165 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,166 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,170 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,171 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/4b5d7860dad24289ad54910ab3327cb6, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 04:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,180 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~108.79 KB/111400, heapSize ~310.25 KB/317696, currentSize=80.51 KB/82440 for 32a2cc8961b71f647281a29aaf4ac2d5 in 443ms, sequenceid=50, compaction requested=false 2023-04-28 04:55:40,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:40,196 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,196 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:40,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:40,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,196 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,197 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,197 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:40,197 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,197 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,197 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:40,197 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:40,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,197 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,198 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 04:55:40,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:40,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:40,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:40,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:40,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:40,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:40,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:40,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657800265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657800266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657800268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657800271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657800276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,298 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:40,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:40,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:40,336 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:40,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,338 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:40,338 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:40,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,343 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,344 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,344 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,344 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:40,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:40,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:40,366 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:40,366 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,370 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:40,371 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:40,371 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:40,375 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,376 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:40,376 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:40,376 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657800375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,377 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:40,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:40,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:40,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:40,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657800375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,385 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,386 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657800382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:40,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e17084e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:40,392 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:40,392 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,396 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,396 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:40,396 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:40,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:40,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:40,407 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:40,407 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,407 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,408 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:40,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:40,413 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:40,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:40,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:40,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:40,413 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:40,426 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,426 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,427 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,428 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,449 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,449 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,449 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,449 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,449 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,449 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,449 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:40,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:40,450 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,450 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,453 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:40,453 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:40,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:40,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,554 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:40,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:40,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:40,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:40,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:40,564 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:40,564 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:40,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:40,566 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,566 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,586 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,586 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657800573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657800573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,588 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,588 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:40,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657800586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:40,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:40,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657800591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657800591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:40,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:40,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:40,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:40,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:40,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:40,606 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,606 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:40,606 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:40,606 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,609 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:40,609 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:40,609 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,609 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:40,609 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:40,617 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,618 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,618 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:40,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d62692[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:40,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:40,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,628 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:40,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:40,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:40,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:40,641 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:40,641 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,641 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,641 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:40,651 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:40,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:40,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:40,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:40,652 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:40,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:40,659 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,659 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,659 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:40,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:40,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:40,681 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:40,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,681 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:40,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce 2023-04-28 04:55:40,782 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:40,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:40,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:40,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:40,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:40,801 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:40,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,802 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:40,802 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:40,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:40,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:40,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:40,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:40,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:40,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:40,828 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:40,829 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:40,829 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,829 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:40,830 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:40,830 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:40,830 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:40,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:40,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:40,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:40,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:40,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,838 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,839 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,840 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a378c5c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:40,847 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:40,848 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,859 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,859 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:40,859 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:40,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:40,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:40,870 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,870 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:40,870 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,870 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,874 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-28 04:55:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,877 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-28 04:55:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:40,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:40,879 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:40,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:40,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:40,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:40,879 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:40,891 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,891 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:40,891 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:40,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:40,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657800893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,894 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657800900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:40,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657800903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,929 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,929 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,929 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:40,930 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:40,930 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:40,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:40,930 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:40,930 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,930 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:40,930 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:40,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:40,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:40,930 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:40,933 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:40,933 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:40,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:40,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:41,033 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:41,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:41,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:41,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:41,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:41,047 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:41,047 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:41,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:41,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,059 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,059 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:41,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:41,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:41,063 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:41,063 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:41,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:41,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:41,064 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,064 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:41,064 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:41,064 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:41,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:41,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:41,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:41,079 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,079 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,079 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,079 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cad677[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:41,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:41,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,091 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:41,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:41,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:41,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:41,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:41,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657801093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:41,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657801094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:41,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:41,102 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:41,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:41,128 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:41,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:41,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:41,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:41,128 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:41,143 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,150 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:41,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:41,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,181 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:41,181 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:41,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/8246c8840f074a538b02eeb26e6b4645 2023-04-28 04:55:41,281 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:41,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:41,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:41,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:41,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:41,288 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:41,288 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,289 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:41,289 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:41,289 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,289 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:41,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:41,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:41,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:41,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:41,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:41,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:41,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:41,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,322 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,323 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,323 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:41,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6579eb38[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:41,325 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:41,325 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,333 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,333 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:41,333 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:41,333 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:41,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:41,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:41,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,345 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:41,345 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:41,345 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:41,345 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:41,345 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:41,345 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:41,345 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:41,354 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,354 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,354 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,355 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,376 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:41,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:41,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:41,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,376 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:41,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:41,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:41,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657801400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:41,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657801407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:41,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657801409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,477 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:41,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:41,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:41,483 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:41,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:41,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:41,485 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,485 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,491 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,491 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,491 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:41,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:41,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:41,493 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:41,493 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,494 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:41,494 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:41,494 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:41,494 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,494 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:41,495 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:41,495 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:41,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:41,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:41,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:41,501 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,502 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,502 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:41,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5729b3b3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:41,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:41,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,512 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:41,512 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:41,512 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:41,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:41,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:41,522 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:41,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:41,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:41,527 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:41,527 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:41,527 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:41,527 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:41,527 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:41,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,541 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:41,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:41,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:41,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:41,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:41,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,584 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/93488f29591c49bbbde79aba01f1d7a5 2023-04-28 04:55:41,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:41,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:41,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:41,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce 2023-04-28 04:55:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:41,703 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:41,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:41,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:41,711 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,712 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce, entries=150, sequenceid=68, filesize=11.6 K 2023-04-28 04:55:41,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/8246c8840f074a538b02eeb26e6b4645 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8246c8840f074a538b02eeb26e6b4645 2023-04-28 04:55:41,733 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,733 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:41,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:41,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:41,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:41,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:41,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,747 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:41,751 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:41,751 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:41,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:41,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:41,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:41,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:41,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:41,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:41,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,765 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8246c8840f074a538b02eeb26e6b4645, entries=150, sequenceid=68, filesize=11.6 K 2023-04-28 04:55:41,765 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,766 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a9066[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:41,768 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:41,768 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/93488f29591c49bbbde79aba01f1d7a5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/93488f29591c49bbbde79aba01f1d7a5 2023-04-28 04:55:41,784 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:55:41,785 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:41,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:41,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:41,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:41,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:41,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:41,812 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:41,816 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:41,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:41,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:41,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:41,816 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:41,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/93488f29591c49bbbde79aba01f1d7a5, entries=150, sequenceid=68, filesize=11.6 K 2023-04-28 04:55:41,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1594ms, sequenceid=68, compaction requested=true 2023-04-28 04:55:41,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:41,828 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,828 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,828 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,828 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:41,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,830 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:41,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:41,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:41,844 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,845 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,845 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,845 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,845 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:41,845 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:41,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:41,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:41,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,845 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,845 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,846 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,846 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:41,846 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:41,846 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,846 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:41,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:41,852 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:41,875 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:41,878 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:41,878 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,878 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/d50c18c6024a4d0e8735f206e42092f1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ad3ec3b951f1421f8d52ac09c68ba3b4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8246c8840f074a538b02eeb26e6b4645] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=34.8 K 2023-04-28 04:55:41,879 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:41,879 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:41,879 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:41,880 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d50c18c6024a4d0e8735f206e42092f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682657738755 2023-04-28 04:55:41,881 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/95f4783a63e6409ca27d1e57969903ed, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9f15d1e8d382462182670d60bfc0b6f7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=39.4 K 2023-04-28 04:55:41,881 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ad3ec3b951f1421f8d52ac09c68ba3b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657739081 2023-04-28 04:55:41,881 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 95f4783a63e6409ca27d1e57969903ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682657738755 2023-04-28 04:55:41,888 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8246c8840f074a538b02eeb26e6b4645, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682657739900 2023-04-28 04:55:41,889 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9f15d1e8d382462182670d60bfc0b6f7, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657739069 2023-04-28 04:55:41,895 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3d0b03c3cf6e46c7b4d0bb9f8a5e37ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682657739900 2023-04-28 04:55:41,920 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#9 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:41,946 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:41,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:41,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:41,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:41,956 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:41,956 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:41,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:41,973 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#10 average throughput is 0.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:41,974 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,974 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,986 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,986 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:41,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:41,986 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:41,986 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:41,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:41,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:41,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:41,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:41,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:41,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:41,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:42,004 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:42,004 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:42,005 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 04:55:42,005 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:42,005 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:42,005 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:42,005 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:42,005 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:42,005 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:42,011 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/69840abb92dc478f9895e160efbdcb66 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/69840abb92dc478f9895e160efbdcb66 2023-04-28 04:55:42,051 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into 69840abb92dc478f9895e160efbdcb66(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:42,052 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:42,052 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=13, startTime=1682657741836; duration=0sec 2023-04-28 04:55:42,060 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:42,060 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:42,078 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:42,078 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:42,078 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:42,079 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/fb78e05fc31044448358d5fb9ad73e49, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/4b5d7860dad24289ad54910ab3327cb6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/93488f29591c49bbbde79aba01f1d7a5] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=34.8 K 2023-04-28 04:55:42,080 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting fb78e05fc31044448358d5fb9ad73e49, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682657738755 2023-04-28 04:55:42,081 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4b5d7860dad24289ad54910ab3327cb6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657739081 2023-04-28 04:55:42,082 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 93488f29591c49bbbde79aba01f1d7a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682657739900 2023-04-28 04:55:42,086 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/9e846719d784474eba03d5e920ba4e1e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9e846719d784474eba03d5e920ba4e1e 2023-04-28 04:55:42,115 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#12 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:42,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:42,122 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 9e846719d784474eba03d5e920ba4e1e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:42,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:42,124 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:42,125 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=13, startTime=1682657741826; duration=0sec 2023-04-28 04:55:42,125 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:42,173 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/abfd24e509bc46bbb10a34f43683f7dd as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/abfd24e509bc46bbb10a34f43683f7dd 2023-04-28 04:55:42,187 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into abfd24e509bc46bbb10a34f43683f7dd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:42,187 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:42,187 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=13, startTime=1682657741836; duration=0sec 2023-04-28 04:55:42,187 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:42,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657802222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657802225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,272 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-28 04:55:42,273 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-28 04:55:42,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657802339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657802340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657802409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657802417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657802421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,467 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/f5f96d73a51a421987183d7d8470e988 2023-04-28 04:55:42,521 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/8088ec9c0ee24ecdbea98c104b52d5a9 2023-04-28 04:55:42,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657802543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657802544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657802850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:42,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:42,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657802851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,001 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/c272f43486a241e89814e4c93f81c09a 2023-04-28 04:55:43,014 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/f5f96d73a51a421987183d7d8470e988 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/f5f96d73a51a421987183d7d8470e988 2023-04-28 04:55:43,033 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/f5f96d73a51a421987183d7d8470e988, entries=150, sequenceid=83, filesize=11.6 K 2023-04-28 04:55:43,038 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/8088ec9c0ee24ecdbea98c104b52d5a9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8088ec9c0ee24ecdbea98c104b52d5a9 2023-04-28 04:55:43,055 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8088ec9c0ee24ecdbea98c104b52d5a9, entries=150, sequenceid=83, filesize=11.6 K 2023-04-28 04:55:43,057 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/c272f43486a241e89814e4c93f81c09a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c272f43486a241e89814e4c93f81c09a 2023-04-28 04:55:43,073 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c272f43486a241e89814e4c93f81c09a, entries=150, sequenceid=83, filesize=11.6 K 2023-04-28 04:55:43,075 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1070ms, sequenceid=83, compaction requested=false 2023-04-28 04:55:43,076 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:43,077 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:43,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:43,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:43,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:43,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5486975a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:43,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:43,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,101 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:43,101 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,101 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:43,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:43,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:43,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:43,112 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:43,116 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:43,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:43,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:43,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:43,117 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:43,122 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,122 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,122 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:43,123 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,124 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:43,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:43,133 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,133 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,133 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:43,133 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:43,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:43,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:43,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:43,234 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:43,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:43,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:43,257 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:43,258 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:43,259 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:43,259 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:43,259 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,260 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,270 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:43,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,272 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,272 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:43,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:43,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:43,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:43,274 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:43,274 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:43,275 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:43,277 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:43,277 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,277 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:43,277 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:43,277 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 04:55:43,277 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:43,278 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:43,278 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:43,278 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:43,278 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:43,278 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:43,332 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/31087e5c455f4c309ff0cc1e99b15463 2023-04-28 04:55:43,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:43,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:43,389 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c538e8efeecc4e41940d81a65d327999 2023-04-28 04:55:43,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657803410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657803433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,467 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/94a94ac404ba4bda86d7a1e45b28ee8d 2023-04-28 04:55:43,479 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/31087e5c455f4c309ff0cc1e99b15463 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/31087e5c455f4c309ff0cc1e99b15463 2023-04-28 04:55:43,488 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/31087e5c455f4c309ff0cc1e99b15463, entries=150, sequenceid=103, filesize=11.6 K 2023-04-28 04:55:43,492 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c538e8efeecc4e41940d81a65d327999 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c538e8efeecc4e41940d81a65d327999 2023-04-28 04:55:43,502 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c538e8efeecc4e41940d81a65d327999, entries=150, sequenceid=103, filesize=11.6 K 2023-04-28 04:55:43,507 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/94a94ac404ba4bda86d7a1e45b28ee8d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/94a94ac404ba4bda86d7a1e45b28ee8d 2023-04-28 04:55:43,526 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/94a94ac404ba4bda86d7a1e45b28ee8d, entries=150, sequenceid=103, filesize=11.6 K 2023-04-28 04:55:43,531 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 32a2cc8961b71f647281a29aaf4ac2d5 in 254ms, sequenceid=103, compaction requested=true 2023-04-28 04:55:43,531 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:43,531 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:43,536 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:43,536 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,536 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:43,536 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:43,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:43,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 04:55:43,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:43,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:43,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:43,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:43,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:43,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:43,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,544 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,544 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e0c96d9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:43,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:43,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,554 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:43,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,555 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:43,555 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:43,555 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:43,564 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:43,564 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:43,564 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:43,569 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:43,572 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:43,572 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:43,572 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:43,573 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:43,573 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:43,591 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,591 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,591 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:43,593 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,596 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657803602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657803603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:43,607 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:43,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:43,607 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,607 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,607 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,607 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,607 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:43,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:43,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:43,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:43,609 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:43,609 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:43,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:43,709 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:43,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:43,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657803711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657803711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:43,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:43,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:43,723 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:43,723 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:43,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:43,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:43,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,763 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:43,764 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,764 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:43,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:43,766 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:43,766 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:43,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:43,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:43,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:43,770 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,770 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:43,770 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:43,770 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:43,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:43,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:43,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:43,786 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64221680[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:43,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:43,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,796 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,797 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:43,797 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:43,797 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:43,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:43,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:43,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:43,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:43,819 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:43,819 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:43,819 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:43,819 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:43,819 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:43,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,839 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,840 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:43,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,849 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:43,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:43,849 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:43,849 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:43,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:43,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657803914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:43,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657803915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,961 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:43,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:43,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:43,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:43,965 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:43,965 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:43,966 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:43,966 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:43,966 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,966 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,975 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:43,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:43,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:43,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:43,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:43,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:43,977 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:43,977 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:43,978 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,978 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:43,978 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:43,978 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:43,978 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:43,985 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,985 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:43,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:43,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:43,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315e13c2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:43,988 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:43,988 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,992 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/ed4d892f003c46eabbe27416c8677b4a 2023-04-28 04:55:43,996 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:43,996 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:43,996 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:43,996 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:43,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:43,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:44,007 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,007 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,007 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,008 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:44,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:44,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/b3fbc0a037cb4a0794dc733b7b4944c0 2023-04-28 04:55:44,026 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:44,026 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:44,026 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:44,026 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:44,026 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:44,039 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,039 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,039 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,041 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:44,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,049 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,049 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,049 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:44,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,151 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:44,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:44,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:44,155 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:44,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,156 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:44,156 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:44,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,202 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:44,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:44,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:44,204 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:44,204 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:44,205 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,205 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:44,205 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:44,205 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:44,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:44,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657804218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657804218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,227 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,227 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,228 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,228 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51027ef4[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:44,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:44,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,238 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,238 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:44,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:44,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:44,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:44,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,248 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,248 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,249 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:44,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:44,252 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:44,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:44,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:44,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:44,252 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:44,260 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,260 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,260 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,265 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,279 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,279 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,279 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:44,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:44,279 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,281 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,281 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,281 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,281 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,281 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,281 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,281 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,381 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:44,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:44,387 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:44,387 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:44,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:44,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657804417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,421 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:44,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657804425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,427 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:44,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,433 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:44,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:44,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:44,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:44,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:44,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:44,441 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,441 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:44,441 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:44,441 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:44,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:44,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:44,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:44,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657804444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,447 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:44,459 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,459 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,460 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,460 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:44,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b228053[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:44,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:44,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/d9e9ef1022c44573b523b2af8081c8af 2023-04-28 04:55:44,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/ed4d892f003c46eabbe27416c8677b4a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ed4d892f003c46eabbe27416c8677b4a 2023-04-28 04:55:44,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ed4d892f003c46eabbe27416c8677b4a, entries=150, sequenceid=119, filesize=11.6 K 2023-04-28 04:55:44,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/b3fbc0a037cb4a0794dc733b7b4944c0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/b3fbc0a037cb4a0794dc733b7b4944c0 2023-04-28 04:55:44,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/b3fbc0a037cb4a0794dc733b7b4944c0, entries=100, sequenceid=119, filesize=9.3 K 2023-04-28 04:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/d9e9ef1022c44573b523b2af8081c8af as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d9e9ef1022c44573b523b2af8081c8af 2023-04-28 04:55:44,496 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:44,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:44,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:44,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:44,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d9e9ef1022c44573b523b2af8081c8af, entries=100, sequenceid=119, filesize=9.3 K 2023-04-28 04:55:44,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 32a2cc8961b71f647281a29aaf4ac2d5 in 971ms, sequenceid=119, compaction requested=true 2023-04-28 04:55:44,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:44,510 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:44,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:44,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:44,510 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:44,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:44,512 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:44,512 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,512 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:44,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,514 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47643 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:44,514 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:44,514 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,514 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9e846719d784474eba03d5e920ba4e1e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/f5f96d73a51a421987183d7d8470e988, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/31087e5c455f4c309ff0cc1e99b15463, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ed4d892f003c46eabbe27416c8677b4a] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=46.5 K 2023-04-28 04:55:44,514 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:44,515 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:44,515 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,515 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/69840abb92dc478f9895e160efbdcb66, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8088ec9c0ee24ecdbea98c104b52d5a9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c538e8efeecc4e41940d81a65d327999, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/b3fbc0a037cb4a0794dc733b7b4944c0] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=44.2 K 2023-04-28 04:55:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,523 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9e846719d784474eba03d5e920ba4e1e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682657739900 2023-04-28 04:55:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,524 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 69840abb92dc478f9895e160efbdcb66, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682657739900 2023-04-28 04:55:44,524 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f5f96d73a51a421987183d7d8470e988, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682657740257 2023-04-28 04:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,525 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8088ec9c0ee24ecdbea98c104b52d5a9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682657740257 2023-04-28 04:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,525 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 31087e5c455f4c309ff0cc1e99b15463, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682657742213 2023-04-28 04:55:44,526 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c538e8efeecc4e41940d81a65d327999, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682657742213 2023-04-28 04:55:44,526 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ed4d892f003c46eabbe27416c8677b4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682657743392 2023-04-28 04:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,526 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b3fbc0a037cb4a0794dc733b7b4944c0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682657743392 2023-04-28 04:55:44,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:44,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:44,535 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:44,535 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:44,535 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:44,535 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:44,535 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:44,547 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:44,549 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:44,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,605 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,606 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,638 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,638 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,638 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:44,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:44,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:44,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:44,640 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:44,638 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,639 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:44,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 04:55:44,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:44,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:44,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:44,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:44,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:44,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:44,740 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:44,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:44,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:44,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:44,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:44,750 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:44,750 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/6939e406ea7444288f2f34713a2ba1fc 2023-04-28 04:55:44,751 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:44,751 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:44,752 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,752 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,764 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657804764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657804765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,772 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:44,774 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:44,774 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,774 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:44,774 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:44,775 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:44,775 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,775 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:44,775 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:44,775 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,775 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:44,775 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:44,785 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,788 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,788 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,788 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:44,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fc009a1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:44,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:44,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,796 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,797 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:44,797 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:44,797 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:44,797 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,797 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:44,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/2aad44304636435d92c6c433a77a6c4b 2023-04-28 04:55:44,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:44,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:44,812 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:44,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:44,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:44,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:44,812 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:44,818 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,818 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,818 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,839 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,839 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:44,839 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:44,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:44,839 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,840 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,840 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,840 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:44,840 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,839 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:44,840 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:44,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:44,840 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:44,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/343acae1dfb44db89cc4252ae1ef03f1 2023-04-28 04:55:44,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/6939e406ea7444288f2f34713a2ba1fc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6939e406ea7444288f2f34713a2ba1fc 2023-04-28 04:55:44,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6939e406ea7444288f2f34713a2ba1fc, entries=200, sequenceid=136, filesize=14.0 K 2023-04-28 04:55:44,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/2aad44304636435d92c6c433a77a6c4b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/2aad44304636435d92c6c433a77a6c4b 2023-04-28 04:55:44,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657804870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/2aad44304636435d92c6c433a77a6c4b, entries=150, sequenceid=136, filesize=11.8 K 2023-04-28 04:55:44,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:44,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657804871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/343acae1dfb44db89cc4252ae1ef03f1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/343acae1dfb44db89cc4252ae1ef03f1 2023-04-28 04:55:44,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/343acae1dfb44db89cc4252ae1ef03f1, entries=150, sequenceid=136, filesize=11.8 K 2023-04-28 04:55:44,897 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 32a2cc8961b71f647281a29aaf4ac2d5 in 174ms, sequenceid=136, compaction requested=true 2023-04-28 04:55:44,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:44,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 04:55:44,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-28 04:55:44,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-28 04:55:44,942 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:44,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:44,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:44,953 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:44,953 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,954 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:44,954 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:44,955 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,955 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,988 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,988 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:44,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:44,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:44,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:44,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:44,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:44,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:44,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:44,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:44,998 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:44,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:44,998 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:44,999 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 04:55:44,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:44,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:44,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:44,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:44,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:44,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:45,011 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/4f7bfa1307594a5783de97263117f0e6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4f7bfa1307594a5783de97263117f0e6 2023-04-28 04:55:45,013 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/014efb7e11df436d8d32673321257baf as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/014efb7e11df436d8d32673321257baf 2023-04-28 04:55:45,029 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 4f7bfa1307594a5783de97263117f0e6(size=11.8 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:45,029 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into 014efb7e11df436d8d32673321257baf(size=11.8 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:45,038 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:45,038 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=12, startTime=1682657744510; duration=0sec 2023-04-28 04:55:45,038 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-28 04:55:45,040 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:55:45,040 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:45,043 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=12, startTime=1682657744510; duration=0sec 2023-04-28 04:55:45,045 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-28 04:55:45,049 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57339 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:55:45,050 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/c80020c9bc264eb488950297c0318f5b 2023-04-28 04:55:45,051 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-28 04:55:45,051 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 04:55:45,051 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 04:55:45,051 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:55:45,051 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 04:55:45,051 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:45,052 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:45,056 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/abfd24e509bc46bbb10a34f43683f7dd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c272f43486a241e89814e4c93f81c09a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/94a94ac404ba4bda86d7a1e45b28ee8d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d9e9ef1022c44573b523b2af8081c8af, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/343acae1dfb44db89cc4252ae1ef03f1] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=56.0 K 2023-04-28 04:55:45,056 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 04:55:45,057 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 04:55:45,057 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting abfd24e509bc46bbb10a34f43683f7dd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682657739900 2023-04-28 04:55:45,057 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:55:45,058 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 04:55:45,058 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c272f43486a241e89814e4c93f81c09a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682657740257 2023-04-28 04:55:45,059 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 04:55:45,059 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 04:55:45,059 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:55:45,062 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 94a94ac404ba4bda86d7a1e45b28ee8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682657742213 2023-04-28 04:55:45,063 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d9e9ef1022c44573b523b2af8081c8af, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682657743392 2023-04-28 04:55:45,064 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 343acae1dfb44db89cc4252ae1ef03f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682657743569 2023-04-28 04:55:45,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:45,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:45,106 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:45,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657805129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657805132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,157 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/a5801c64989f45e3bf927db124ea004b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/a5801c64989f45e3bf927db124ea004b 2023-04-28 04:55:45,171 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into a5801c64989f45e3bf927db124ea004b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:45,172 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:45,172 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=11, startTime=1682657744898; duration=0sec 2023-04-28 04:55:45,172 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:45,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657805236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657805237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657805442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657805442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,519 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/577e44d761014885affaab623ce31f6c 2023-04-28 04:55:45,543 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/248aaf45a18f4952aaae7833db392f28 2023-04-28 04:55:45,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/c80020c9bc264eb488950297c0318f5b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/c80020c9bc264eb488950297c0318f5b 2023-04-28 04:55:45,556 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/c80020c9bc264eb488950297c0318f5b, entries=150, sequenceid=151, filesize=11.8 K 2023-04-28 04:55:45,558 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/577e44d761014885affaab623ce31f6c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/577e44d761014885affaab623ce31f6c 2023-04-28 04:55:45,567 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/577e44d761014885affaab623ce31f6c, entries=150, sequenceid=151, filesize=11.8 K 2023-04-28 04:55:45,572 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/248aaf45a18f4952aaae7833db392f28 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/248aaf45a18f4952aaae7833db392f28 2023-04-28 04:55:45,581 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/248aaf45a18f4952aaae7833db392f28, entries=150, sequenceid=151, filesize=11.8 K 2023-04-28 04:55:45,585 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 32a2cc8961b71f647281a29aaf4ac2d5 in 586ms, sequenceid=151, compaction requested=true 2023-04-28 04:55:45,586 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:45,586 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:45,586 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:45,586 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:45,586 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,586 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:45,587 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:45,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:45,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:45,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:45,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:45,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e41cd98[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:45,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:45,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,648 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:45,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:45,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:45,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:45,659 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:45,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:45,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:45,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:45,660 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:45,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:45,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:45,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:45,660 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:45,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:45,672 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:45,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:45,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:45,681 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:45,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:45,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:45,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:45,682 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:45,682 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:45,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 04:55:45,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:45,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:45,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:45,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:45,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:45,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:45,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/1d0398bf946c4b96880cace5609ae404 2023-04-28 04:55:45,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657805770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657805774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,782 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:45,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:45,786 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:45,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:45,789 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:45,789 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:45,789 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,789 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:45,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,817 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:45,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:45,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:45,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:45,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:45,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:45,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:45,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:45,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:45,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:45,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:45,820 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:45,820 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:45,820 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:45,820 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:45,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:45,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:45,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:45,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:45,871 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:45,872 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,872 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c76e97a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:45,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:45,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657805874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:45,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657805877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,890 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:45,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:45,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:45,891 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,891 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:45,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:45,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:45,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:45,998 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:45,998 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:46,001 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:46,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:46,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:46,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:46,001 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:46,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:46,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,047 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:46,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:46,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:46,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,070 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:46,070 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:46,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:46,070 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:46,071 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:46,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:46,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657806081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657806083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,171 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:46,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:46,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:46,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:46,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/400fbb6acdc74c0580ab9d34fc20a6ab 2023-04-28 04:55:46,186 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:46,186 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:46,187 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:46,187 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:46,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,196 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:46,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:46,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:46,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:46,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:46,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:46,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:46,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:46,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:46,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:46,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:46,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:46,217 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,217 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,217 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,217 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:46,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6309c6bf[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:46,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,225 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/8eaaeb6403a548feb5f552d9dc3b8972 2023-04-28 04:55:46,227 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,227 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:46,228 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:46,228 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:46,227 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,228 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:46,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/1d0398bf946c4b96880cace5609ae404 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/1d0398bf946c4b96880cace5609ae404 2023-04-28 04:55:46,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:46,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/1d0398bf946c4b96880cace5609ae404, entries=200, sequenceid=173, filesize=14.1 K 2023-04-28 04:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/400fbb6acdc74c0580ab9d34fc20a6ab as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/400fbb6acdc74c0580ab9d34fc20a6ab 2023-04-28 04:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:46,243 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:46,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:46,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:46,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:46,243 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:46,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:46,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,250 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/400fbb6acdc74c0580ab9d34fc20a6ab, entries=150, sequenceid=173, filesize=11.8 K 2023-04-28 04:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/8eaaeb6403a548feb5f552d9dc3b8972 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eaaeb6403a548feb5f552d9dc3b8972 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:46,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:46,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:46,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:46,259 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:46,260 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:46,259 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,267 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,267 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:46,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eaaeb6403a548feb5f552d9dc3b8972, entries=150, sequenceid=173, filesize=11.8 K 2023-04-28 04:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:46,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 32a2cc8961b71f647281a29aaf4ac2d5 in 522ms, sequenceid=173, compaction requested=true 2023-04-28 04:55:46,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:46,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:46,270 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:46,270 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:46,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:46,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:46,272 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:46,272 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52964 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:46,272 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:46,272 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:46,272 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,272 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,272 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4f7bfa1307594a5783de97263117f0e6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6939e406ea7444288f2f34713a2ba1fc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/c80020c9bc264eb488950297c0318f5b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/1d0398bf946c4b96880cace5609ae404] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=51.7 K 2023-04-28 04:55:46,272 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/014efb7e11df436d8d32673321257baf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/2aad44304636435d92c6c433a77a6c4b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/577e44d761014885affaab623ce31f6c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/400fbb6acdc74c0580ab9d34fc20a6ab] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=47.1 K 2023-04-28 04:55:46,273 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4f7bfa1307594a5783de97263117f0e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682657742213 2023-04-28 04:55:46,275 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 014efb7e11df436d8d32673321257baf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682657742213 2023-04-28 04:55:46,281 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6939e406ea7444288f2f34713a2ba1fc, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682657743548 2023-04-28 04:55:46,281 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 2aad44304636435d92c6c433a77a6c4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682657743569 2023-04-28 04:55:46,282 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c80020c9bc264eb488950297c0318f5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682657744747 2023-04-28 04:55:46,282 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 577e44d761014885affaab623ce31f6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682657744747 2023-04-28 04:55:46,283 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 400fbb6acdc74c0580ab9d34fc20a6ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682657745129 2023-04-28 04:55:46,283 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 1d0398bf946c4b96880cace5609ae404, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682657745099 2023-04-28 04:55:46,309 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:46,311 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#34 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:46,329 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c44c4dbfcd8c485f8ca7d416fc55515f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c44c4dbfcd8c485f8ca7d416fc55515f 2023-04-28 04:55:46,337 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into c44c4dbfcd8c485f8ca7d416fc55515f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:46,337 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:46,337 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=12, startTime=1682657746270; duration=0sec 2023-04-28 04:55:46,341 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:46,341 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:46,344 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:46,344 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:46,344 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,345 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/a5801c64989f45e3bf927db124ea004b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/248aaf45a18f4952aaae7833db392f28, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eaaeb6403a548feb5f552d9dc3b8972] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=35.5 K 2023-04-28 04:55:46,346 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a5801c64989f45e3bf927db124ea004b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682657743569 2023-04-28 04:55:46,347 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 248aaf45a18f4952aaae7833db392f28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682657744747 2023-04-28 04:55:46,348 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8eaaeb6403a548feb5f552d9dc3b8972, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682657745129 2023-04-28 04:55:46,353 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/0feae721e6b84febbe34c0eff33e54cb as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0feae721e6b84febbe34c0eff33e54cb 2023-04-28 04:55:46,360 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#35 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:46,362 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 0feae721e6b84febbe34c0eff33e54cb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:46,364 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:46,364 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=12, startTime=1682657746270; duration=0sec 2023-04-28 04:55:46,364 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:46,367 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:46,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:46,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:46,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:46,373 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:46,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:46,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:46,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:46,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,375 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:46,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 04:55:46,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:46,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:46,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:46,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:46,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:46,426 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,426 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:46,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:46,426 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,426 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:46,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:46,427 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:46,427 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:46,429 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:46,430 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,430 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:46,430 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:46,430 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:46,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:46,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657806452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657806454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,490 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,490 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,491 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,491 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:46,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cbb567e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:46,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:46,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,553 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,553 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:46,553 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,553 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:46,553 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:46,553 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657806555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657806555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,637 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,637 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:46,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:46,641 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:46,641 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:46,641 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:46,641 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:46,641 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:46,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,702 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657806758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:46,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657806758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:46,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:46,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:46,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:46,765 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:46,766 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:46,765 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:46,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:46,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:46,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:46,785 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/0d1d0238f3ec4083a44f3788cddeebf9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/0d1d0238f3ec4083a44f3788cddeebf9 2023-04-28 04:55:46,795 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into 0d1d0238f3ec4083a44f3788cddeebf9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:46,796 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:46,796 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=13, startTime=1682657746270; duration=0sec 2023-04-28 04:55:46,796 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:46,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/51becd6317ee46008b945fa1a264bb22 2023-04-28 04:55:46,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/dba94690c54d46b48b690214e9e442f7 2023-04-28 04:55:46,866 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:46,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:46,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:46,872 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:46,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:46,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:46,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:46,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,923 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,924 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,924 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:46,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:46,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:46,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:46,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:46,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:46,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:46,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:46,929 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,929 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:46,929 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:46,929 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:46,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:46,943 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:46,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:46,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce66d7c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:46,945 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:46,945 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,019 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,019 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,019 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:47,019 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:47,019 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:47,019 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:47,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:47,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43636 deadline: 1682657807061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:47,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43640 deadline: 1682657807067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,092 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:47,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,093 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:47,093 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:47,105 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:47,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:47,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:47,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:47,105 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:47,166 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,167 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,167 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:47,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:47,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/17220c2da1054e1f98e971685035b490 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:47,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,273 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:47,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:47,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:47,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:47,274 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:47,274 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:47,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:47,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/51becd6317ee46008b945fa1a264bb22 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/51becd6317ee46008b945fa1a264bb22 2023-04-28 04:55:47,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/51becd6317ee46008b945fa1a264bb22, entries=200, sequenceid=189, filesize=14.1 K 2023-04-28 04:55:47,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/dba94690c54d46b48b690214e9e442f7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dba94690c54d46b48b690214e9e442f7 2023-04-28 04:55:47,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dba94690c54d46b48b690214e9e442f7, entries=150, sequenceid=189, filesize=11.8 K 2023-04-28 04:55:47,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/17220c2da1054e1f98e971685035b490 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/17220c2da1054e1f98e971685035b490 2023-04-28 04:55:47,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/17220c2da1054e1f98e971685035b490, entries=150, sequenceid=189, filesize=11.8 K 2023-04-28 04:55:47,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 32a2cc8961b71f647281a29aaf4ac2d5 in 916ms, sequenceid=189, compaction requested=false 2023-04-28 04:55:47,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:47,374 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:47,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:47,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:47,378 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:47,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:47,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:47,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:47,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,396 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,396 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:47,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:47,396 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,396 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:47,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:47,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:47,408 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:47,408 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:47,408 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:47,408 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:47,408 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:47,408 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:47,409 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:47,409 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:47,442 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/6805821b97854364a2e179bab7c95d15 2023-04-28 04:55:47,481 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/ea3a07a3a7a1439090b92bbf111f82d8 2023-04-28 04:55:47,528 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/f896a95e5cc64eb28ca0ba4e958161db 2023-04-28 04:55:47,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/6805821b97854364a2e179bab7c95d15 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6805821b97854364a2e179bab7c95d15 2023-04-28 04:55:47,548 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6805821b97854364a2e179bab7c95d15, entries=150, sequenceid=209, filesize=11.8 K 2023-04-28 04:55:47,550 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/ea3a07a3a7a1439090b92bbf111f82d8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ea3a07a3a7a1439090b92bbf111f82d8 2023-04-28 04:55:47,566 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ea3a07a3a7a1439090b92bbf111f82d8, entries=150, sequenceid=209, filesize=11.8 K 2023-04-28 04:55:47,567 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/f896a95e5cc64eb28ca0ba4e958161db as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f896a95e5cc64eb28ca0ba4e958161db 2023-04-28 04:55:47,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:47,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:47,578 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f896a95e5cc64eb28ca0ba4e958161db, entries=150, sequenceid=209, filesize=11.8 K 2023-04-28 04:55:47,579 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=13.42 KB/13740 for 32a2cc8961b71f647281a29aaf4ac2d5 in 170ms, sequenceid=209, compaction requested=true 2023-04-28 04:55:47,579 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:47,579 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:47,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:47,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:47,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:47,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:47,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:55:47,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:47,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:47,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:47,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:47,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:47,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:47,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:47,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d07ca3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:47,598 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:47,598 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:47,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:47,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:47,607 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,607 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:47,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,617 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:47,617 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:47,617 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:47,620 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:47,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:47,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:47,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:47,620 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:47,627 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,627 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,627 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:47,659 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/5844cace2841400e8cd4ba8fab7e7cba 2023-04-28 04:55:47,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:47,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:47,671 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:47,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:47,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,671 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:47,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,671 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,671 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:47,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:47,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657747661, max timestamp=1682657747676Num uniques 859; 2023-04-28 04:55:47,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:55:47,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:47,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657807700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657807702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,771 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:47,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:47,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:47,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:47,775 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:47,775 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:47,775 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:47,775 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:47,776 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,777 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,785 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:47,785 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:47,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:47,787 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:47,787 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:47,787 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:47,787 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:47,788 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:47,796 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a416ad8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:47,798 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:47,798 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:47,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:47,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:47,808 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,808 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:47,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:47,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657807808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:47,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657807809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,817 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:47,817 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,817 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:47,817 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:47,821 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:47,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:47,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:47,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:47,821 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:47,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:47,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,838 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,839 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:47,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:47,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:47,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,849 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:47,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:47,851 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:47,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:47,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:47,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:47,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:47,951 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:47,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:47,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:47,964 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:47,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:47,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:47,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:47,965 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,965 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,974 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,975 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:47,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:47,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:47,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:47,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:47,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:47,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:47,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:47,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:47,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:47,986 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:47,986 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:47,986 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:47,986 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:47,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:47,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:47,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:47,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:47,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,995 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:47,996 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:47,996 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c4fc548[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:47,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:47,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:48,006 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:48,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:48,006 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,006 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:48,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:48,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:48,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:48,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657808010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657808010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,017 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,017 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:48,017 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:48,017 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:48,018 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:48,019 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:48,019 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:48,019 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:48,019 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:48,027 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,028 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,028 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,028 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:48,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:48,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:48,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:48,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:48,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,106 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:48,106 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:48,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:48,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:48,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:48,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:48,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/54450447cfc64f48b7d652cbb5c8612e 2023-04-28 04:55:48,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/d48e3453d31247e7a0084febecc19cad 2023-04-28 04:55:48,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/5844cace2841400e8cd4ba8fab7e7cba as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5844cace2841400e8cd4ba8fab7e7cba 2023-04-28 04:55:48,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5844cace2841400e8cd4ba8fab7e7cba, entries=300, sequenceid=220, filesize=18.8 K 2023-04-28 04:55:48,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/54450447cfc64f48b7d652cbb5c8612e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/54450447cfc64f48b7d652cbb5c8612e 2023-04-28 04:55:48,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/54450447cfc64f48b7d652cbb5c8612e, entries=150, sequenceid=220, filesize=11.8 K 2023-04-28 04:55:48,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/d48e3453d31247e7a0084febecc19cad as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d48e3453d31247e7a0084febecc19cad 2023-04-28 04:55:48,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d48e3453d31247e7a0084febecc19cad, entries=150, sequenceid=220, filesize=11.8 K 2023-04-28 04:55:48,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 32a2cc8961b71f647281a29aaf4ac2d5 in 566ms, sequenceid=220, compaction requested=true 2023-04-28 04:55:48,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:48,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:48,157 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:48,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:48,157 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:48,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:48,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:48,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:48,162 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:48,162 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c44c4dbfcd8c485f8ca7d416fc55515f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dba94690c54d46b48b690214e9e442f7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ea3a07a3a7a1439090b92bbf111f82d8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/54450447cfc64f48b7d652cbb5c8612e] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=47.4 K 2023-04-28 04:55:48,162 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 58080 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:48,162 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:48,162 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:48,162 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0feae721e6b84febbe34c0eff33e54cb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/51becd6317ee46008b945fa1a264bb22, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6805821b97854364a2e179bab7c95d15, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5844cace2841400e8cd4ba8fab7e7cba] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=56.7 K 2023-04-28 04:55:48,163 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0feae721e6b84febbe34c0eff33e54cb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682657745129 2023-04-28 04:55:48,163 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c44c4dbfcd8c485f8ca7d416fc55515f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682657745129 2023-04-28 04:55:48,163 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 51becd6317ee46008b945fa1a264bb22, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682657745764 2023-04-28 04:55:48,165 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6805821b97854364a2e179bab7c95d15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682657746414 2023-04-28 04:55:48,165 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting dba94690c54d46b48b690214e9e442f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682657745764 2023-04-28 04:55:48,166 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5844cace2841400e8cd4ba8fab7e7cba, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682657747575 2023-04-28 04:55:48,166 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ea3a07a3a7a1439090b92bbf111f82d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682657746414 2023-04-28 04:55:48,167 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 54450447cfc64f48b7d652cbb5c8612e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682657747586 2023-04-28 04:55:48,181 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:48,184 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:48,206 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:48,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:48,209 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:48,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:48,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:48,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:48,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:48,217 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:48,217 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:48,217 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:48,217 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:48,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:48,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:48,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:48,219 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:48,219 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:48,219 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:48,219 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:48,219 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:48,219 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:48,220 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=125.23 KB heapSize=366.80 KB 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:48,220 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:48,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:48,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657808330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657808336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657808437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657808442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657808452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,458 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657808455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,459 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:48,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657808485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,490 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8222 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:55:48,611 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/92b098c55b9b4c1ab2bfab37c8a88eee as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/92b098c55b9b4c1ab2bfab37c8a88eee 2023-04-28 04:55:48,613 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/104350af7f344724977503f0a3d4160b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/104350af7f344724977503f0a3d4160b 2023-04-28 04:55:48,617 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 92b098c55b9b4c1ab2bfab37c8a88eee(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:48,617 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:48,617 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=12, startTime=1682657748157; duration=0sec 2023-04-28 04:55:48,617 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:48,618 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:48,619 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into 104350af7f344724977503f0a3d4160b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:48,619 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:48,619 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=12, startTime=1682657748157; duration=0sec 2023-04-28 04:55:48,619 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:48,619 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:48,619 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:48,619 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:48,620 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/0d1d0238f3ec4083a44f3788cddeebf9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/17220c2da1054e1f98e971685035b490, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f896a95e5cc64eb28ca0ba4e958161db, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d48e3453d31247e7a0084febecc19cad] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=47.4 K 2023-04-28 04:55:48,620 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0d1d0238f3ec4083a44f3788cddeebf9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682657745129 2023-04-28 04:55:48,620 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 17220c2da1054e1f98e971685035b490, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682657745764 2023-04-28 04:55:48,621 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f896a95e5cc64eb28ca0ba4e958161db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682657746414 2023-04-28 04:55:48,621 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d48e3453d31247e7a0084febecc19cad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682657747586 2023-04-28 04:55:48,629 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:48,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657808640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,651 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/828f3b051d7946fd85dc8fe01f2bcd23 2023-04-28 04:55:48,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657808649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657808945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:48,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:48,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657808958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,041 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/32471823ccc8425eac085d8dcfdd11a7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/32471823ccc8425eac085d8dcfdd11a7 2023-04-28 04:55:49,047 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into 32471823ccc8425eac085d8dcfdd11a7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:49,048 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:49,048 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=12, startTime=1682657748157; duration=0sec 2023-04-28 04:55:49,048 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:49,072 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/073d27c3c7cc47da9c86bf7d0fd3c38b 2023-04-28 04:55:49,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:49,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657809454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:49,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657809465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,486 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/99c45d324b6f45d986fb1e32c9c9f210 2023-04-28 04:55:49,493 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/828f3b051d7946fd85dc8fe01f2bcd23 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/828f3b051d7946fd85dc8fe01f2bcd23 2023-04-28 04:55:49,498 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/828f3b051d7946fd85dc8fe01f2bcd23, entries=150, sequenceid=246, filesize=11.8 K 2023-04-28 04:55:49,500 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/073d27c3c7cc47da9c86bf7d0fd3c38b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/073d27c3c7cc47da9c86bf7d0fd3c38b 2023-04-28 04:55:49,505 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/073d27c3c7cc47da9c86bf7d0fd3c38b, entries=150, sequenceid=246, filesize=11.8 K 2023-04-28 04:55:49,506 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/99c45d324b6f45d986fb1e32c9c9f210 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/99c45d324b6f45d986fb1e32c9c9f210 2023-04-28 04:55:49,512 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/99c45d324b6f45d986fb1e32c9c9f210, entries=150, sequenceid=246, filesize=11.8 K 2023-04-28 04:55:49,513 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~366.75 KB/375552, currentSize=53.67 KB/54960 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1292ms, sequenceid=246, compaction requested=false 2023-04-28 04:55:49,513 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:49,513 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:49,513 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:49,513 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:49,513 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,513 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:49,513 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:49,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:49,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:49,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:49,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@135dc430[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:49,545 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:49,545 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,553 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:49,554 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:49,554 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:49,554 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:49,564 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,564 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:49,564 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:49,564 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:49,566 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:49,566 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:49,566 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:49,566 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:49,566 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:49,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:49,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:49,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:49,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:49,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:49,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:49,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:49,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:49,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:49,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:49,586 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:49,586 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,586 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,586 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:49,586 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:49,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:49,586 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:49,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:49,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:49,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:49,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:49,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:49,690 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:49,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:49,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:49,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:49,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:49,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:49,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:49,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:49,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:49,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:49,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:49,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:49,702 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:49,702 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:49,702 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:49,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:49,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:49,703 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:49,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:49,719 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/0204fa0edd1d497a88a811af1de0fdc3 2023-04-28 04:55:50,143 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/daeb37dde7e546f8b97fb6aa97e84b44 2023-04-28 04:55:50,180 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/bc1ecab3f12e486caa5bf88843e7fe6f 2023-04-28 04:55:50,190 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/0204fa0edd1d497a88a811af1de0fdc3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0204fa0edd1d497a88a811af1de0fdc3 2023-04-28 04:55:50,197 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0204fa0edd1d497a88a811af1de0fdc3, entries=150, sequenceid=260, filesize=11.8 K 2023-04-28 04:55:50,198 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/daeb37dde7e546f8b97fb6aa97e84b44 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/daeb37dde7e546f8b97fb6aa97e84b44 2023-04-28 04:55:50,208 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/daeb37dde7e546f8b97fb6aa97e84b44, entries=150, sequenceid=260, filesize=11.8 K 2023-04-28 04:55:50,209 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/bc1ecab3f12e486caa5bf88843e7fe6f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/bc1ecab3f12e486caa5bf88843e7fe6f 2023-04-28 04:55:50,241 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/bc1ecab3f12e486caa5bf88843e7fe6f, entries=150, sequenceid=260, filesize=11.8 K 2023-04-28 04:55:50,242 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 32a2cc8961b71f647281a29aaf4ac2d5 in 539ms, sequenceid=260, compaction requested=true 2023-04-28 04:55:50,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:50,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:50,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:50,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:50,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:50,269 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374c879c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:50,271 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:50,271 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,280 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:50,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:50,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:50,290 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:50,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:50,294 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:50,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:50,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:50,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:50,294 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:50,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:50,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,311 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:50,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:50,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:50,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,312 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,312 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,312 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,312 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,312 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,412 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:50,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:50,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:50,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:50,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:50,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:50,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,436 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,436 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:50,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:50,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:50,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:50,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:50,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:50,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:50,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:50,444 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,444 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:50,445 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:50,445 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:50,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:50,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:50,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:50,457 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:50,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@365a2c07[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:50,459 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:50,459 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:50,469 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:50,469 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:50,469 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:50,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:55:50,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:50,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:50,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,480 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:50,480 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,480 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:50,483 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:50,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:50,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:50,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:50,483 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:50,501 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,501 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,501 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,501 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,511 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,511 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,511 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:50,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:50,511 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:50,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,512 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:50,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,512 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:50,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,512 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657750508, max timestamp=1682657750528Num uniques 859; 2023-04-28 04:55:50,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:55:50,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657750508, max timestamp=1682657750528Num uniques 859; 2023-04-28 04:55:50,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:55:50,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657750508, max timestamp=1682657750528Num uniques 859; 2023-04-28 04:55:50,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:55:50,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657810558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:50,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657810560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:50,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:50,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:50,616 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:50,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:50,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:50,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:50,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657810661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:50,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657810663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,688 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,688 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,688 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,688 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:50,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:50,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:50,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:50,689 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:50,689 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:50,689 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:50,690 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,690 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:50,690 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:50,690 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:50,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:50,711 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,712 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,712 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444dda01[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:50,713 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:50,713 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,722 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,722 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:50,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:50,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:50,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:50,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:50,734 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:50,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:50,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:50,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:50,734 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:50,743 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,743 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,743 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,743 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,744 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:50,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:50,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,753 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:50,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:50,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:50,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:50,860 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:50,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:50,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:50,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:50,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:50,869 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,869 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657810866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,870 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,870 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:50,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657810869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:50,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:50,871 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,871 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:50,871 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:50,871 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:50,871 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:50,872 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,872 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:50,872 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:50,879 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,880 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,880 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,880 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394cfafc[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:50,882 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:50,882 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,890 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:50,890 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:50,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:50,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:50,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/5b34bcbd414e4105baeceb46ebcff119 2023-04-28 04:55:50,902 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,902 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:50,902 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,902 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:50,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:50,914 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:50,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:50,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:50,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:50,914 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:50,922 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,922 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,922 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:50,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:50,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:50,923 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,924 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:50,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:50,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:50,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:50,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:50,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:50,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:50,933 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:50,933 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:50,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,033 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:51,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:51,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:51,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:51,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:51,048 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:51,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:51,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:51,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:51,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:51,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:51,110 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:51,110 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,110 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:51,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:51,112 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:51,112 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:51,113 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:51,113 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,113 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:51,113 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:51,113 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,113 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:51,113 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:51,143 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:51,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aeffa2e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:51,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:51,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:51,154 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:51,154 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:51,154 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:51,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:51,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:51,166 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:51,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:51,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:51,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:51,166 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:51,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:51,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657811171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,174 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,174 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:51,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657811174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,176 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,185 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:51,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,186 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:51,186 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:51,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,286 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:51,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:51,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:51,292 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:51,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:51,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:51,293 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,293 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:51,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:51,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:51,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:51,303 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:51,303 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,303 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:51,303 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:51,303 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:51,303 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,303 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:51,303 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:51,303 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:51,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:51,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:51,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:51,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/9f6c5af6bad145d59af54cacb0745888 2023-04-28 04:55:51,341 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:51,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@db2d960[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:51,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:51,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,364 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:51,364 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,364 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:51,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:51,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:51,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,374 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:51,374 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,374 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:51,376 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:51,376 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:51,376 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:51,376 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:51,377 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:51,385 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,385 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,385 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,385 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:51,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:51,395 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,396 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,396 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:51,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:51,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,396 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,396 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,396 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,496 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:51,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:51,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:51,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:51,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:51,499 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:51,499 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,499 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:51,499 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:51,499 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,500 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,522 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:51,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:51,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:51,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:51,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:51,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:51,527 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,527 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:51,527 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:51,527 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:51,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:51,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:51,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:51,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,532 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,532 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,532 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:51,534 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27859705[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:51,534 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:51,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:51,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:51,543 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:51,553 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:51,553 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,553 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,553 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:51,555 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:51,555 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:51,555 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:51,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:51,556 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:51,564 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,564 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,564 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,564 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,574 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:51,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:51,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,575 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,575 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:51,575 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,575 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:51,575 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,675 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:51,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:51,678 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:51,678 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,678 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:51,678 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:51,679 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,679 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657811676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:51,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657811677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,691 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,691 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:51,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:51,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:51,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:51,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:51,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:51,693 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,694 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:51,694 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:51,694 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:51,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:51,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:51,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:51,700 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79c00213[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:51,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:51,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,711 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:51,711 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,711 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:51,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:51,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:51,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:51,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:51,725 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:51,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:51,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:51,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:51,725 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:51,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/8eff271131d94c45ad6c7fbf0a564d2f 2023-04-28 04:55:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,743 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/5b34bcbd414e4105baeceb46ebcff119 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5b34bcbd414e4105baeceb46ebcff119 2023-04-28 04:55:51,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5b34bcbd414e4105baeceb46ebcff119, entries=150, sequenceid=271, filesize=11.9 K 2023-04-28 04:55:51,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:51,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:51,753 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:51,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,754 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:51,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/9f6c5af6bad145d59af54cacb0745888 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/9f6c5af6bad145d59af54cacb0745888 2023-04-28 04:55:51,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:51,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/9f6c5af6bad145d59af54cacb0745888, entries=150, sequenceid=271, filesize=11.9 K 2023-04-28 04:55:51,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/8eff271131d94c45ad6c7fbf0a564d2f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eff271131d94c45ad6c7fbf0a564d2f 2023-04-28 04:55:51,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eff271131d94c45ad6c7fbf0a564d2f, entries=150, sequenceid=271, filesize=11.9 K 2023-04-28 04:55:51,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1305ms, sequenceid=271, compaction requested=true 2023-04-28 04:55:51,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:51,777 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:51,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:51,777 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:51,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:51,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:51,779 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48806 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:51,779 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:51,779 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48806 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:51,779 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,779 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:51,779 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,780 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/104350af7f344724977503f0a3d4160b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/073d27c3c7cc47da9c86bf7d0fd3c38b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/daeb37dde7e546f8b97fb6aa97e84b44, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/9f6c5af6bad145d59af54cacb0745888] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=47.7 K 2023-04-28 04:55:51,780 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/92b098c55b9b4c1ab2bfab37c8a88eee, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/828f3b051d7946fd85dc8fe01f2bcd23, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0204fa0edd1d497a88a811af1de0fdc3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5b34bcbd414e4105baeceb46ebcff119] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=47.7 K 2023-04-28 04:55:51,780 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 104350af7f344724977503f0a3d4160b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682657747586 2023-04-28 04:55:51,780 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 92b098c55b9b4c1ab2bfab37c8a88eee, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682657747586 2023-04-28 04:55:51,781 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 828f3b051d7946fd85dc8fe01f2bcd23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682657747691 2023-04-28 04:55:51,781 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 073d27c3c7cc47da9c86bf7d0fd3c38b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682657747691 2023-04-28 04:55:51,781 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0204fa0edd1d497a88a811af1de0fdc3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682657748322 2023-04-28 04:55:51,782 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting daeb37dde7e546f8b97fb6aa97e84b44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682657748322 2023-04-28 04:55:51,782 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5b34bcbd414e4105baeceb46ebcff119, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657750466 2023-04-28 04:55:51,782 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9f6c5af6bad145d59af54cacb0745888, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657750466 2023-04-28 04:55:51,795 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:51,796 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:51,831 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/48691b4ef0564b82a0777e80704230e5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/48691b4ef0564b82a0777e80704230e5 2023-04-28 04:55:51,834 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/97e9f5f924b54ddcacbd528e782ea6a4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/97e9f5f924b54ddcacbd528e782ea6a4 2023-04-28 04:55:51,840 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into 48691b4ef0564b82a0777e80704230e5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:51,840 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:51,840 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=12, startTime=1682657751777; duration=0sec 2023-04-28 04:55:51,840 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:51,841 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:51,841 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 97e9f5f924b54ddcacbd528e782ea6a4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:51,841 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:51,841 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=12, startTime=1682657751777; duration=0sec 2023-04-28 04:55:51,841 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:51,842 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48806 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:51,842 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:51,842 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,842 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/32471823ccc8425eac085d8dcfdd11a7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/99c45d324b6f45d986fb1e32c9c9f210, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/bc1ecab3f12e486caa5bf88843e7fe6f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eff271131d94c45ad6c7fbf0a564d2f] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=47.7 K 2023-04-28 04:55:51,843 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 32471823ccc8425eac085d8dcfdd11a7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682657747586 2023-04-28 04:55:51,843 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 99c45d324b6f45d986fb1e32c9c9f210, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682657747691 2023-04-28 04:55:51,843 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting bc1ecab3f12e486caa5bf88843e7fe6f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682657748322 2023-04-28 04:55:51,844 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8eff271131d94c45ad6c7fbf0a564d2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657750466 2023-04-28 04:55:51,851 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#59 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:51,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:51,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:51,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:51,873 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:51,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,874 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:51,874 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:51,874 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,874 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,879 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/3e3c3975554340b29910f11d1c0e87ed as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3e3c3975554340b29910f11d1c0e87ed 2023-04-28 04:55:51,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,880 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:51,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:51,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:51,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:51,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:51,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:51,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:51,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:51,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:51,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:51,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:51,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:51,882 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-28 04:55:51,883 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:51,883 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:51,883 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:51,883 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:51,883 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:51,883 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:51,887 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into 3e3c3975554340b29910f11d1c0e87ed(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:51,887 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:51,887 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=12, startTime=1682657751778; duration=0sec 2023-04-28 04:55:51,887 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:51,893 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/7975595e11194b3486c5332164d6d495 2023-04-28 04:55:51,904 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/46b8679e96bd4abb982d56006192c331 2023-04-28 04:55:51,914 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/881c8e47a43b4c41b62083b0223ddd8e 2023-04-28 04:55:51,920 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/7975595e11194b3486c5332164d6d495 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/7975595e11194b3486c5332164d6d495 2023-04-28 04:55:51,925 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/7975595e11194b3486c5332164d6d495, entries=150, sequenceid=307, filesize=11.9 K 2023-04-28 04:55:51,927 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/46b8679e96bd4abb982d56006192c331 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/46b8679e96bd4abb982d56006192c331 2023-04-28 04:55:51,935 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/46b8679e96bd4abb982d56006192c331, entries=150, sequenceid=307, filesize=11.9 K 2023-04-28 04:55:51,936 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/881c8e47a43b4c41b62083b0223ddd8e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/881c8e47a43b4c41b62083b0223ddd8e 2023-04-28 04:55:51,942 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/881c8e47a43b4c41b62083b0223ddd8e, entries=150, sequenceid=307, filesize=11.9 K 2023-04-28 04:55:51,943 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=0 B/0 for 32a2cc8961b71f647281a29aaf4ac2d5 in 60ms, sequenceid=307, compaction requested=false 2023-04-28 04:55:51,943 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:51,943 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:51,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:51,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:51,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:51,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:51,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,953 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:51,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:51,955 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cebb98f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:51,955 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,964 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:51,964 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:51,964 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:51,964 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:51,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:51,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:52,006 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:52,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:52,008 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:52,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:52,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:52,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:52,008 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:52,027 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,027 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,027 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,027 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,028 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,038 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:52,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:52,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,038 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,038 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:52,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:52,038 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,038 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,038 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:52,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,138 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:52,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:52,142 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:52,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:52,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:52,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,153 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:52,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:52,155 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,155 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:52,155 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:52,155 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:52,155 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:52,156 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,156 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:52,156 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:52,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:52,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770914ab[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:52,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:52,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,174 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:52,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:52,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:52,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:52,185 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:52,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,185 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,185 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:52,188 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:52,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:52,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:52,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:52,188 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:52,195 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,195 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,195 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:52,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:52,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:52,206 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,207 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:52,207 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:52,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,307 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:52,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:52,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:52,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:52,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:52,310 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:52,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:52,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:52,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,322 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,322 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,322 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:52,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:52,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:52,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:52,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:52,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:52,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:52,332 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,332 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,332 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,332 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbf88f0[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:52,334 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:52,334 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,343 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:52,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:52,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:52,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:52,353 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:52,353 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,353 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,353 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:52,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:52,356 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:52,356 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:52,356 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:52,356 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:52,356 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:52,364 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,364 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,364 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,365 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,366 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:52,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:52,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:52,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:52,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:52,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,475 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:52,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:52,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:52,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:52,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:52,479 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:52,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:52,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:52,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,490 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,490 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,490 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,490 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:52,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:52,491 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:52,492 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,492 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:52,492 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:52,492 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:52,492 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:52,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:52,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@516cdee[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:52,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,511 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,511 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:52,511 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:52,511 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:52,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:52,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:52,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:52,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:52,531 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:52,531 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:52,531 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:52,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:52,532 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,571 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,572 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,594 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,594 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,594 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,595 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:52,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,595 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:52,595 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:52,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:52,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,695 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:52,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:55:52,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:52,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:52,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:52,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:52,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:52,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:52,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:52,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:52,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:52,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:52,703 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:52,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:52,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:52,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/4a643f933af1460682962f4efa0b2afe 2023-04-28 04:55:52,711 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,711 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,712 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,712 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:52,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:52,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:52,713 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:52,714 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,714 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:52,714 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:52,714 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:52,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:52,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:52,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,722 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34002b02[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:52,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:52,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657752718, max timestamp=1682657752731Num uniques 859; 2023-04-28 04:55:52,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:55:52,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657752718, max timestamp=1682657752731Num uniques 859; 2023-04-28 04:55:52,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:55:52,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657752718, max timestamp=1682657752731Num uniques 859; 2023-04-28 04:55:52,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:55:52,734 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:52,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:52,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:52,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,734 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:52,743 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:52,743 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,743 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,743 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:52,747 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:52,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:52,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:52,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:52,747 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:52,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,754 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,755 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657812758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,764 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:52,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,764 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:52,764 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:52,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657812759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,864 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:52,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:52,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657812862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:52,867 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:52,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:52,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:52,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:52,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657812866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,879 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:52,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:52,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:52,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:52,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:52,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:52,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:52,890 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,890 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b13cd65[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:52,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:52,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,900 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,901 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,901 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:52,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:52,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:52,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:52,911 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:52,911 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,911 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,911 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:52,913 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:52,913 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:52,913 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:52,913 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:52,913 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:52,921 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,921 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:52,922 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:52,922 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,923 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:52,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:52,933 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,933 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:52,933 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,034 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:53,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:53,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:53,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:53,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:53,036 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:53,037 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,037 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:53,037 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:53,037 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,037 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:53,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:53,050 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,050 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:53,050 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:53,050 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:53,050 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,051 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:53,051 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:53,058 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:53,059 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6b2cff[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:53,061 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:53,061 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,069 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:53,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:53,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:53,070 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,070 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:53,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657813068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:53,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657813070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,084 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:53,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:53,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:53,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:53,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:53,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:53,100 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,101 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,101 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,101 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,103 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:53,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:53,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,139 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,140 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,140 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:53,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/cb09787fbc95495aa11b635abe6a4127 2023-04-28 04:55:53,240 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:53,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:53,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:53,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:53,243 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:53,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:53,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:53,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,248 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,248 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,248 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:53,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:53,249 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:53,249 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,249 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:53,256 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,256 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:53,256 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:53,256 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:53,256 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:53,269 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38a1e9c1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:53,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:53,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,279 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:53,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:53,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:53,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:53,290 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:53,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:53,292 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:53,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:53,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:53,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:53,292 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:53,300 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,301 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,311 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:53,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,312 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:53,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:53,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657813374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657813374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,412 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:53,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:53,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:53,419 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:53,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:53,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:53,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,427 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,427 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,427 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,427 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:53,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:53,431 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,431 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:53,431 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:53,431 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:53,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:53,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,439 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,439 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:53,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:53,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dcc9dec[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:53,448 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,448 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:53,448 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,449 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:53,449 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:53,449 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:53,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:53,458 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,459 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:53,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:53,462 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:53,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:53,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:53,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:53,462 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:53,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:53,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:53,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:53,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/8e38295e0de74e5b98c36a2a69ebb135 2023-04-28 04:55:53,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/4a643f933af1460682962f4efa0b2afe as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4a643f933af1460682962f4efa0b2afe 2023-04-28 04:55:53,580 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:53,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:53,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:53,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4a643f933af1460682962f4efa0b2afe, entries=250, sequenceid=322, filesize=16.7 K 2023-04-28 04:55:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:53,585 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:53,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:53,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:53,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/cb09787fbc95495aa11b635abe6a4127 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/cb09787fbc95495aa11b635abe6a4127 2023-04-28 04:55:53,586 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,586 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,595 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:53,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:53,598 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:53,598 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,598 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:53,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:53,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:53,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/cb09787fbc95495aa11b635abe6a4127, entries=150, sequenceid=322, filesize=11.9 K 2023-04-28 04:55:53,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/8e38295e0de74e5b98c36a2a69ebb135 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8e38295e0de74e5b98c36a2a69ebb135 2023-04-28 04:55:53,604 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,604 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:53,604 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:53,604 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,604 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:53,604 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:53,604 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8e38295e0de74e5b98c36a2a69ebb135, entries=150, sequenceid=322, filesize=11.9 K 2023-04-28 04:55:53,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 32a2cc8961b71f647281a29aaf4ac2d5 in 920ms, sequenceid=322, compaction requested=true 2023-04-28 04:55:53,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:53,616 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:53,616 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:53,616 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:53,616 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:53,616 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:53,617 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:53,617 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:53,617 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,617 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/48691b4ef0564b82a0777e80704230e5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/46b8679e96bd4abb982d56006192c331, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/cb09787fbc95495aa11b635abe6a4127] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.3 K 2023-04-28 04:55:53,618 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42086 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:53,618 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:53,618 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,618 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 48691b4ef0564b82a0777e80704230e5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657750466 2023-04-28 04:55:53,618 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/97e9f5f924b54ddcacbd528e782ea6a4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/7975595e11194b3486c5332164d6d495, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4a643f933af1460682962f4efa0b2afe] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=41.1 K 2023-04-28 04:55:53,619 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 46b8679e96bd4abb982d56006192c331, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682657750532 2023-04-28 04:55:53,619 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 97e9f5f924b54ddcacbd528e782ea6a4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657750466 2023-04-28 04:55:53,619 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting cb09787fbc95495aa11b635abe6a4127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682657752691 2023-04-28 04:55:53,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:53,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:53,620 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7975595e11194b3486c5332164d6d495, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682657750532 2023-04-28 04:55:53,621 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4a643f933af1460682962f4efa0b2afe, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682657752691 2023-04-28 04:55:53,627 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a111d5[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:53,630 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:53,630 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,637 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:53,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:53,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:53,638 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,638 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:53,648 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,648 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:53,648 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,649 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:53,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:53,653 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:53,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:53,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:53,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:53,653 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:53,657 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:53,659 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,659 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:53,659 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:53,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:53,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,698 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,698 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:53,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:53,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:53,698 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:53,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:53,713 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c9dab642941d4602be0638dd355fd961 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c9dab642941d4602be0638dd355fd961 2023-04-28 04:55:53,715 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/a8853160416546ecb59402d717a242a2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a8853160416546ecb59402d717a242a2 2023-04-28 04:55:53,727 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into c9dab642941d4602be0638dd355fd961(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:53,727 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:53,727 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=13, startTime=1682657753616; duration=0sec 2023-04-28 04:55:53,727 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:53,727 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:53,728 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:53,728 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:53,729 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,729 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3e3c3975554340b29910f11d1c0e87ed, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/881c8e47a43b4c41b62083b0223ddd8e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8e38295e0de74e5b98c36a2a69ebb135] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.3 K 2023-04-28 04:55:53,729 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 3e3c3975554340b29910f11d1c0e87ed, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657750466 2023-04-28 04:55:53,730 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 881c8e47a43b4c41b62083b0223ddd8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682657750532 2023-04-28 04:55:53,730 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into a8853160416546ecb59402d717a242a2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:53,731 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:53,731 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=13, startTime=1682657753616; duration=0sec 2023-04-28 04:55:53,731 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:53,731 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8e38295e0de74e5b98c36a2a69ebb135, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682657752691 2023-04-28 04:55:53,749 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:53,786 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/f67565650e9d4f34a22b0e3f6291b88d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f67565650e9d4f34a22b0e3f6291b88d 2023-04-28 04:55:53,794 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into f67565650e9d4f34a22b0e3f6291b88d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:53,795 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:53,795 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=13, startTime=1682657753616; duration=0sec 2023-04-28 04:55:53,795 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:53,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:53,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:53,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:53,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:53,802 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:53,802 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,802 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:53,802 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:53,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:53,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,818 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:53,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:53,818 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:53,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:53,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:53,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:53,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:53,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:53,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:53,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:53,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:53,824 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-28 04:55:53,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:53,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:53,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:53,825 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:53,825 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:53,825 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:53,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:53,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657813896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:53,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:53,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657813900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657814001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657814003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657814204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657814208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,236 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/48ee65c3365743f5b33eabfe3cd72c55 2023-04-28 04:55:54,283 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c171551d939147f8ae25b1a7f7f946f0 2023-04-28 04:55:54,325 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/c25c5bc0e6174b1ea256568925fdff7e 2023-04-28 04:55:54,337 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/48ee65c3365743f5b33eabfe3cd72c55 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/48ee65c3365743f5b33eabfe3cd72c55 2023-04-28 04:55:54,346 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/48ee65c3365743f5b33eabfe3cd72c55, entries=150, sequenceid=359, filesize=11.9 K 2023-04-28 04:55:54,347 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c171551d939147f8ae25b1a7f7f946f0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c171551d939147f8ae25b1a7f7f946f0 2023-04-28 04:55:54,361 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c171551d939147f8ae25b1a7f7f946f0, entries=150, sequenceid=359, filesize=11.9 K 2023-04-28 04:55:54,363 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/c25c5bc0e6174b1ea256568925fdff7e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c25c5bc0e6174b1ea256568925fdff7e 2023-04-28 04:55:54,367 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c25c5bc0e6174b1ea256568925fdff7e, entries=150, sequenceid=359, filesize=11.9 K 2023-04-28 04:55:54,368 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 32a2cc8961b71f647281a29aaf4ac2d5 in 544ms, sequenceid=359, compaction requested=false 2023-04-28 04:55:54,368 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:54,368 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:54,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:54,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:54,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:54,433 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4affdaea[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:54,435 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:54,435 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,447 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:54,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:54,447 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,447 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:54,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:54,458 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,459 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:54,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:54,461 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:54,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:54,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:54,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:54,461 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:54,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,469 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:54,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:54,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:54,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:54,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:54,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:54,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:54,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:54,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:54,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:54,480 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:54,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:54,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 04:55:54,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:54,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:54,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:54,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:54,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:54,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:54,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657814544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657814545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,580 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:54,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:54,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:54,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:54,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:54,583 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:54,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:54,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:54,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:54,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:54,595 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:54,596 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,596 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:54,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:54,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:54,597 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,597 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:54,597 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:54,597 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:54,597 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:54,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7da7cd95[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:54,608 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:54,608 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,616 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:54,616 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:54,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,617 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:54,616 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:54,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:54,627 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:54,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:54,635 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:54,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:54,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:54,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:54,635 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:54,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657814648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,655 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657814653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:54,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:54,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:54,669 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:54,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,668 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:54,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:54,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:54,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:54,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:54,769 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:54,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:54,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:54,773 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:54,773 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:54,773 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:54,773 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:54,774 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,774 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,785 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:54,785 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:54,785 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,785 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:54,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:54,786 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,786 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:54,786 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:54,786 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:54,786 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:54,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:54,787 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:54,795 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146d1c30[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:54,797 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:54,797 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:54,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:54,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:54,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:54,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,816 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,816 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:54,817 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:54,819 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:54,819 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:54,819 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:54,819 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:54,819 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:54,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:54,827 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,828 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,837 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:54,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:54,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:54,838 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:54,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:54,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:54,838 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:54,838 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:54,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657814853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657814862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/18075098efb14e86b67175411b466db0 2023-04-28 04:55:54,938 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:54,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:54,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:54,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:54,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:54,941 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:54,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:54,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:54,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:54,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,953 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:54,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:54,953 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,953 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:54,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:54,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:54,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:54,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:54,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:54,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:54,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:54,954 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,955 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:54,955 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:54,955 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:54,955 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:54,955 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:54,955 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,955 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:54,955 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:54,994 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:54,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:54,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d298d1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:54,996 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:54,996 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,016 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,016 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,016 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:55,016 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:55,017 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:55,017 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:55,027 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:55,027 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,027 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:55,028 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:55,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:55,032 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:55,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:55,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:55,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:55,032 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:55,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,040 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,048 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,048 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:55,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:55,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,148 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:55,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:55,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:55,151 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:55,151 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,151 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:55,151 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:55,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:55,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657815157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,164 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,164 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,164 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:55,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:55,165 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,165 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:55,165 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:55,165 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:55,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:55,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:55,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:55,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657815164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,174 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,174 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:55,176 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ce3a2b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:55,176 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,185 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,185 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:55,185 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:55,185 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:55,185 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,185 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:55,195 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:55,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:55,198 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:55,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:55,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:55,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:55,198 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:55,208 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,208 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,208 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,213 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,227 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:55,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:55,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:55,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:55,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:55,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:55,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:55,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:55,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:55,330 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:55,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:55,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:55,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c4f4dd8e481b42fca469fe13eaa415ca 2023-04-28 04:55:55,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:55,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:55,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:55,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:55,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:55,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:55,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:55,353 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,353 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,353 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,353 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:55,355 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:55,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@734692c2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:55,355 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,436 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:55,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:55,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:55,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:55,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:55,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:55,509 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:55,511 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:55,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:55,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:55,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:55,511 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:55,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,523 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:55,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:55,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:55,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:55,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,633 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:55,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:55,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:55,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:55,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:55,635 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:55,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:55,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:55,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,648 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:55,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:55,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:55,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:55,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:55,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:55,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:55,649 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:55,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:55,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:55,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:55,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:55,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:55,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,658 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f368c06[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:55,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:55,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:55,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657815663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:55,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657815668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:55,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:55,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:55,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:55,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/69b76e81aef44f708ab2463341ac77ef 2023-04-28 04:55:55,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/18075098efb14e86b67175411b466db0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/18075098efb14e86b67175411b466db0 2023-04-28 04:55:55,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/18075098efb14e86b67175411b466db0, entries=150, sequenceid=371, filesize=11.9 K 2023-04-28 04:55:55,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/c4f4dd8e481b42fca469fe13eaa415ca as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c4f4dd8e481b42fca469fe13eaa415ca 2023-04-28 04:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:55,756 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:55,756 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:55,756 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:55,756 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:55,756 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:55,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,764 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,764 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:55,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c4f4dd8e481b42fca469fe13eaa415ca, entries=150, sequenceid=371, filesize=11.9 K 2023-04-28 04:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,766 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/69b76e81aef44f708ab2463341ac77ef as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/69b76e81aef44f708ab2463341ac77ef 2023-04-28 04:55:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/69b76e81aef44f708ab2463341ac77ef, entries=150, sequenceid=371, filesize=11.9 K 2023-04-28 04:55:55,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1262ms, sequenceid=371, compaction requested=true 2023-04-28 04:55:55,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:55,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:55,773 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:55,773 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:55,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:55,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:55,774 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:55,775 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:55,775 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:55,774 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,775 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:55,775 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a8853160416546ecb59402d717a242a2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/48ee65c3365743f5b33eabfe3cd72c55, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/18075098efb14e86b67175411b466db0] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.4 K 2023-04-28 04:55:55,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:55,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:55,775 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,775 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:55,775 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:55,775 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:55,775 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting a8853160416546ecb59402d717a242a2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682657752691 2023-04-28 04:55:55,776 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:55,776 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,776 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c9dab642941d4602be0638dd355fd961, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c171551d939147f8ae25b1a7f7f946f0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c4f4dd8e481b42fca469fe13eaa415ca] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.4 K 2023-04-28 04:55:55,778 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,778 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:55,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:55,778 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 48ee65c3365743f5b33eabfe3cd72c55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682657752750 2023-04-28 04:55:55,778 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c9dab642941d4602be0638dd355fd961, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682657752691 2023-04-28 04:55:55,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:55,779 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 18075098efb14e86b67175411b466db0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682657753881 2023-04-28 04:55:55,779 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c171551d939147f8ae25b1a7f7f946f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682657752750 2023-04-28 04:55:55,779 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c4f4dd8e481b42fca469fe13eaa415ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682657753881 2023-04-28 04:55:55,786 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:55,790 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:55,815 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/dd5a9e2032164dca8fd0524f78b84ba4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dd5a9e2032164dca8fd0524f78b84ba4 2023-04-28 04:55:55,822 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into dd5a9e2032164dca8fd0524f78b84ba4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:55,822 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:55,822 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=13, startTime=1682657755773; duration=0sec 2023-04-28 04:55:55,822 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:55,822 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:55,823 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:55,823 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:55,823 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,823 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f67565650e9d4f34a22b0e3f6291b88d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c25c5bc0e6174b1ea256568925fdff7e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/69b76e81aef44f708ab2463341ac77ef] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.4 K 2023-04-28 04:55:55,824 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting f67565650e9d4f34a22b0e3f6291b88d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682657752691 2023-04-28 04:55:55,824 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c25c5bc0e6174b1ea256568925fdff7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682657752750 2023-04-28 04:55:55,825 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 69b76e81aef44f708ab2463341ac77ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682657753881 2023-04-28 04:55:55,841 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#77 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:55,853 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/d2c65d28332c49c999a4975dcb21850d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d2c65d28332c49c999a4975dcb21850d 2023-04-28 04:55:55,858 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into d2c65d28332c49c999a4975dcb21850d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:55,858 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:55,859 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=13, startTime=1682657755773; duration=0sec 2023-04-28 04:55:55,859 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:55,875 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:55,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:55,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:55,879 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:55,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:55,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:55,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,890 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:55,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:55,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:55,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:55,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:55,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:55,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:55,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:55,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:55,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:55,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:55,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:55,892 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:55,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:55,893 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:55,900 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/805ce57e9c264de7b5b6af46dba9b1a4 2023-04-28 04:55:55,911 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/a6c604c6174946c3b49c6b00f490fd04 2023-04-28 04:55:56,219 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/508fd7bd6f604dfab706a6d2311dac2e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/508fd7bd6f604dfab706a6d2311dac2e 2023-04-28 04:55:56,225 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 508fd7bd6f604dfab706a6d2311dac2e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:56,225 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:56,225 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=13, startTime=1682657755773; duration=0sec 2023-04-28 04:55:56,225 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:56,322 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/62aa041d699744768e2b8818f4691fea 2023-04-28 04:55:56,327 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/805ce57e9c264de7b5b6af46dba9b1a4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/805ce57e9c264de7b5b6af46dba9b1a4 2023-04-28 04:55:56,332 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/805ce57e9c264de7b5b6af46dba9b1a4, entries=150, sequenceid=393, filesize=11.9 K 2023-04-28 04:55:56,333 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/a6c604c6174946c3b49c6b00f490fd04 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/a6c604c6174946c3b49c6b00f490fd04 2023-04-28 04:55:56,338 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/a6c604c6174946c3b49c6b00f490fd04, entries=150, sequenceid=393, filesize=11.9 K 2023-04-28 04:55:56,339 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/62aa041d699744768e2b8818f4691fea as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/62aa041d699744768e2b8818f4691fea 2023-04-28 04:55:56,344 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/62aa041d699744768e2b8818f4691fea, entries=150, sequenceid=393, filesize=11.9 K 2023-04-28 04:55:56,345 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 32a2cc8961b71f647281a29aaf4ac2d5 in 453ms, sequenceid=393, compaction requested=false 2023-04-28 04:55:56,345 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:56,345 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:56,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:56,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:56,345 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:56,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,394 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,395 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,395 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:56,396 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb29921[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:56,396 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,406 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,406 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:56,406 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:56,406 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,406 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:56,406 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:56,416 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:56,416 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,416 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,416 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:56,421 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:56,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:56,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:56,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:56,422 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:56,427 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,427 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,427 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,428 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,430 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,437 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:56,437 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,437 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,437 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,440 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:56,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:56,440 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:56,440 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:56,540 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:56,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:56,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:56,543 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:56,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:56,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:56,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:56,544 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,544 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,553 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:56,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:56,562 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,562 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:56,562 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:56,562 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:56,562 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:56,574 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,574 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eece79e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:56,576 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:56,576 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,585 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,585 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:56,585 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:56,585 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:56,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:56,595 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:56,597 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:56,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:56,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:56,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:56,597 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:56,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,606 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,606 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,607 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:56,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:56,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:56,617 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,617 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:56,617 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:56,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:56,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:55:56,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:56,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:56,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:56,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:56,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:56,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:56,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657756701, max timestamp=1682657756712Num uniques 859; 2023-04-28 04:55:56,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:55:56,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657756701, max timestamp=1682657756712Num uniques 859; 2023-04-28 04:55:56,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:55:56,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657756701, max timestamp=1682657756712Num uniques 859; 2023-04-28 04:55:56,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:55:56,717 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:56,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:56,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:56,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:56,720 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:56,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:56,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:56,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:56,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:56,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:56,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:56,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:56,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:56,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:56,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:56,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:56,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657816736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657816736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,742 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,742 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,743 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,743 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:56,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@197bbe03[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:56,744 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:56,744 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:56,753 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,753 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:56,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:56,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:56,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:56,796 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:56,798 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:56,798 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:56,798 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:56,798 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:56,798 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:56,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,818 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,827 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:56,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:56,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,827 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:56,827 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:56,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657816840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:56,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657816842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,927 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:56,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:56,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:56,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:56,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:56,930 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:56,930 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:56,930 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:56,930 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:56,930 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,930 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,942 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,943 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,943 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:56,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:56,944 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,944 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:56,944 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:56,944 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:56,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:56,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,953 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,953 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,953 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:56,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22165796[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:56,954 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:56,955 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,963 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,964 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:56,964 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:56,964 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:56,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:56,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:56,974 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:56,976 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:56,976 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:56,976 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:56,976 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:56,976 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:56,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:56,985 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,985 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:56,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,986 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:56,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:56,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:56,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:56,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:56,995 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:56,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:56,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:56,995 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:56,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:56,996 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,996 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:56,996 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:56,996 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:56,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:57,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657817047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:57,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657817047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,096 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:57,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:57,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:57,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:57,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/ccfde3d8ba56439d9f0a84b33157e4fe 2023-04-28 04:55:57,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:57,106 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:57,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:57,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:57,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,111 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,111 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,112 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,112 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:57,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:57,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:57,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:57,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:57,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:57,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:57,115 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,115 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:57,115 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:57,115 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:57,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:57,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:57,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:57,121 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,121 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,122 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,122 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:57,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29e6b964[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:57,123 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:57,123 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,132 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,132 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:57,132 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:57,132 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:57,132 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,132 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:57,142 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:57,142 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,142 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:57,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:57,145 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:57,145 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:57,145 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:57,145 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:57,145 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:57,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,154 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,154 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,164 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:57,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:57,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:57,165 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:57,165 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:57,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,265 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:57,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:57,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:57,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:57,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:57,268 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:57,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:57,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:57,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:57,280 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:57,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:57,283 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,283 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:57,283 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:57,283 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:57,283 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:57,290 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:57,291 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@100f6bd1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:57,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,300 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,301 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:57,301 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:57,301 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:57,311 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:57,311 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,311 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,311 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:57,313 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:57,313 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:57,313 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:57,313 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:57,313 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:57,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,324 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,325 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657817349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657817351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:57,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:57,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:57,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:57,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:57,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:57,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:57,463 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:57,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:57,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:57,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:57,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:57,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:57,481 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,481 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:57,481 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:57,481 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:57,481 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:57,490 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,490 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,490 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,490 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:57,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54405475[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:57,491 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:57,491 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,501 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,501 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:57,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:57,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:57,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:57,511 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:57,511 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,511 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,511 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:57,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:57,513 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:57,513 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:57,513 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:57,513 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:57,513 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:57,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,523 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:57,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:57,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:57,532 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:57,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/8ed2f3f70ec34e61995bef9286f3fc87 2023-04-28 04:55:57,633 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:57,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:57,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:57,635 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:57,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:57,635 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:57,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,648 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:57,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:57,649 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,649 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:57,649 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:57,649 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:57,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:57,658 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,659 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:57,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:57,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@447a138c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:57,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:57,669 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:57,669 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:57,669 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:57,679 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:57,679 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,679 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,679 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:57,682 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:57,682 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:57,682 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:57,682 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:57,682 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:57,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,691 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,700 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,700 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,700 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,700 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:57,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:57,700 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:57,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:57,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:57,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:57,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:57,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:57,804 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:57,804 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,804 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:57,804 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:57,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,816 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,816 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:57,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:57,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:57,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:57,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:57,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:57,818 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:57,818 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,818 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:57,819 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:57,819 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:57,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:57,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:57,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:57,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:57,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,827 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,828 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,828 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58cbbffe[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:57,829 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:57,829 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,837 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,837 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:57,837 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,837 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:57,837 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:57,837 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:57,848 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:57,848 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,848 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,848 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:57,849 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:57,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:57,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:57,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:57,850 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:57,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:57,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657817853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:57,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657817854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:57,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:57,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,859 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,864 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:57,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:57,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:57,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:57,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:57,896 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:57,897 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:57,897 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:57,897 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:57,897 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,897 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:57,897 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:57,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:57,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:57,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/9a261064c551443ca2ee849f1043f81c 2023-04-28 04:55:57,972 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/ccfde3d8ba56439d9f0a84b33157e4fe as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ccfde3d8ba56439d9f0a84b33157e4fe 2023-04-28 04:55:57,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ccfde3d8ba56439d9f0a84b33157e4fe, entries=150, sequenceid=405, filesize=11.9 K 2023-04-28 04:55:57,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/8ed2f3f70ec34e61995bef9286f3fc87 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8ed2f3f70ec34e61995bef9286f3fc87 2023-04-28 04:55:57,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8ed2f3f70ec34e61995bef9286f3fc87, entries=150, sequenceid=405, filesize=11.9 K 2023-04-28 04:55:57,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/9a261064c551443ca2ee849f1043f81c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/9a261064c551443ca2ee849f1043f81c 2023-04-28 04:55:57,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:57,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:58,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:58,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/9a261064c551443ca2ee849f1043f81c, entries=150, sequenceid=405, filesize=11.9 K 2023-04-28 04:55:58,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1320ms, sequenceid=405, compaction requested=true 2023-04-28 04:55:58,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:58,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:58,002 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:58,002 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:58,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:58,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:58,004 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:58,004 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:58,004 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:58,004 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:58,004 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,004 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,004 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dd5a9e2032164dca8fd0524f78b84ba4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/a6c604c6174946c3b49c6b00f490fd04, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8ed2f3f70ec34e61995bef9286f3fc87] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.5 K 2023-04-28 04:55:58,004 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/508fd7bd6f604dfab706a6d2311dac2e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/805ce57e9c264de7b5b6af46dba9b1a4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ccfde3d8ba56439d9f0a84b33157e4fe] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=36.5 K 2023-04-28 04:55:58,005 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting dd5a9e2032164dca8fd0524f78b84ba4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682657753881 2023-04-28 04:55:58,005 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 508fd7bd6f604dfab706a6d2311dac2e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682657753881 2023-04-28 04:55:58,005 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a6c604c6174946c3b49c6b00f490fd04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682657754539 2023-04-28 04:55:58,005 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 805ce57e9c264de7b5b6af46dba9b1a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682657754539 2023-04-28 04:55:58,005 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8ed2f3f70ec34e61995bef9286f3fc87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682657756678 2023-04-28 04:55:58,006 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ccfde3d8ba56439d9f0a84b33157e4fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682657756678 2023-04-28 04:55:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:58,008 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:58,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:58,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:58,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:58,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,027 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,028 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#84 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:58,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:58,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:58,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:58,045 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:58,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:58,076 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:58,076 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:58,076 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:58,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:58,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:58,100 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,100 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:58,100 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-28 04:55:58,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:58,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:58,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:58,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:58,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:58,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:55:58,118 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/ddd880ccccc74398933ff78ad2f492b4 2023-04-28 04:55:58,131 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/302bd6bcb78541e9a95e7c0dfa2a7895 2023-04-28 04:55:58,154 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/cda4457585a346418151905041a70101 2023-04-28 04:55:58,159 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/ddd880ccccc74398933ff78ad2f492b4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ddd880ccccc74398933ff78ad2f492b4 2023-04-28 04:55:58,168 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ddd880ccccc74398933ff78ad2f492b4, entries=150, sequenceid=440, filesize=11.9 K 2023-04-28 04:55:58,170 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/302bd6bcb78541e9a95e7c0dfa2a7895 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/302bd6bcb78541e9a95e7c0dfa2a7895 2023-04-28 04:55:58,177 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/302bd6bcb78541e9a95e7c0dfa2a7895, entries=150, sequenceid=440, filesize=11.9 K 2023-04-28 04:55:58,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/cda4457585a346418151905041a70101 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/cda4457585a346418151905041a70101 2023-04-28 04:55:58,186 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/cda4457585a346418151905041a70101, entries=150, sequenceid=440, filesize=11.9 K 2023-04-28 04:55:58,187 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=0 B/0 for 32a2cc8961b71f647281a29aaf4ac2d5 in 87ms, sequenceid=440, compaction requested=true 2023-04-28 04:55:58,187 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:58,187 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,187 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:58,187 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:58,187 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,188 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:58,188 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:58,196 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,196 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,196 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,196 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337106d5[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:58,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:58,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,209 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:58,209 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:58,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:58,209 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:58,216 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:58,216 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,216 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,216 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:58,221 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:58,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:58,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:58,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:58,222 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:58,227 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,227 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,227 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,227 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,237 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:58,237 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:58,237 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:58,237 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,238 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:58,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:58,237 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:58,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:58,238 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:58,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:58,338 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:58,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:58,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:58,342 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:58,342 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:58,342 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:58,342 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:58,342 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,342 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,353 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,353 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:58,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:58,353 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,354 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:58,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:58,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:58,355 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:58,355 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:58,355 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:58,359 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:58,360 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,360 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:58,360 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:58,360 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,360 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:58,361 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:58,361 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:58,361 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,361 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:58,361 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:58,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,374 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,374 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:58,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c40f32[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:58,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:58,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,476 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:58,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:58,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:58,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:58,493 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/da9fb6df05f04648b41b1bdcdcf98eaf as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/da9fb6df05f04648b41b1bdcdcf98eaf 2023-04-28 04:55:58,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:55:58,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:55:58,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:55:58,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:58,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:55:58,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:58,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:55:58,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:55:58,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:58,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,507 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into da9fb6df05f04648b41b1bdcdcf98eaf(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:58,507 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:58,507 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=13, startTime=1682657758002; duration=0sec 2023-04-28 04:55:58,508 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:58,508 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:55:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:58,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:58,508 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:58,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:58,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:58,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:58,508 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:58,511 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/916236f2e585448e9507ff2f4092f44d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/916236f2e585448e9507ff2f4092f44d 2023-04-28 04:55:58,521 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:55:58,521 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/C is initiating minor compaction (all files) 2023-04-28 04:55:58,521 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/C in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,521 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d2c65d28332c49c999a4975dcb21850d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/62aa041d699744768e2b8818f4691fea, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/9a261064c551443ca2ee849f1043f81c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/cda4457585a346418151905041a70101] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=48.4 K 2023-04-28 04:55:58,522 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,522 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:58,522 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,522 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d2c65d28332c49c999a4975dcb21850d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682657753881 2023-04-28 04:55:58,523 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 62aa041d699744768e2b8818f4691fea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682657754539 2023-04-28 04:55:58,523 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9a261064c551443ca2ee849f1043f81c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682657756678 2023-04-28 04:55:58,524 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting cda4457585a346418151905041a70101, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1682657756727 2023-04-28 04:55:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,531 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,533 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 916236f2e585448e9507ff2f4092f44d(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:58,533 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:58,533 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=13, startTime=1682657758002; duration=0sec 2023-04-28 04:55:58,533 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:58,541 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#C#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:58,542 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,542 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:58,542 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,542 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:58,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:58,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:58,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:58,543 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:58,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,543 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:58,543 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:58,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,543 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:58,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657758537, max timestamp=1682657758551Num uniques 859; 2023-04-28 04:55:58,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:55:58,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657758537, max timestamp=1682657758551Num uniques 859; 2023-04-28 04:55:58,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:55:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657758537, max timestamp=1682657758551Num uniques 859; 2023-04-28 04:55:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:55:58,580 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/3a3dec2ffe644820a12d67eeb9017a9c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3a3dec2ffe644820a12d67eeb9017a9c 2023-04-28 04:55:58,592 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/C of 32a2cc8961b71f647281a29aaf4ac2d5 into 3a3dec2ffe644820a12d67eeb9017a9c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:55:58,593 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:58,593 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/C, priority=12, startTime=1682657758002; duration=0sec 2023-04-28 04:55:58,593 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:55:58,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657818603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657818606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657818608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,643 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:58,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:58,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:55:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:55:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:55:58,652 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:55:58,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:58,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:55:58,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:55:58,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,658 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,658 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:58,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:58,659 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,659 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:55:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:55:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:55:58,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:55:58,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:55:58,660 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:55:58,660 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:55:58,660 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:55:58,664 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:55:58,664 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. started... 2023-04-28 04:55:58,664 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. as already flushing 2023-04-28 04:55:58,664 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:55:58,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:55:58,669 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,669 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,669 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,669 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:55:58,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71368675[Count = 0] remaining members to acquire global barrier 2023-04-28 04:55:58,671 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:55:58,671 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,679 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,679 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:55:58,680 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:55:58,680 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:55:58,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:55:58,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:58,690 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,695 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bcd59bb to 127.0.0.1:52932 2023-04-28 04:55:58,695 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:55:58,695 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38ebb9f9 to 127.0.0.1:52932 2023-04-28 04:55:58,695 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:55:58,696 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13112432 to 127.0.0.1:52932 2023-04-28 04:55:58,696 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:55:58,696 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:55:58,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:55:58,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:55:58,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:55:58,696 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:55:58,703 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dcc1cc1 to 127.0.0.1:52932 2023-04-28 04:55:58,703 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:55:58,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657818709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657818709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657818713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:58,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,732 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:55:58,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:58,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:55:58,733 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,771 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:55:58,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:55:58,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:55:58,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:55:58,771 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:55:58,771 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:55:58,771 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:55:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:55:58,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:55:58,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:55:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:55:58,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43636 deadline: 1682657818858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:43640 deadline: 1682657818860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,871 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:55:58,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:55:58,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657818911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657818912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657818915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:58,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/9400b783e55f4e93a7591a78bee19819 2023-04-28 04:55:59,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657819214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:59,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:59,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657819215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:59,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:59,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657819218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:59,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/56e23336cfab4c11b22b054a199e6ae9 2023-04-28 04:55:59,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:59,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43676 deadline: 1682657819716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:59,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43626 deadline: 1682657819719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:59,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:55:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:43662 deadline: 1682657819722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:55:59,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/ba95d38932a840a2a1684537fbedad3e 2023-04-28 04:55:59,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/9400b783e55f4e93a7591a78bee19819 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9400b783e55f4e93a7591a78bee19819 2023-04-28 04:55:59,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9400b783e55f4e93a7591a78bee19819, entries=100, sequenceid=451, filesize=9.5 K 2023-04-28 04:55:59,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/56e23336cfab4c11b22b054a199e6ae9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/56e23336cfab4c11b22b054a199e6ae9 2023-04-28 04:55:59,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/56e23336cfab4c11b22b054a199e6ae9, entries=100, sequenceid=451, filesize=9.5 K 2023-04-28 04:55:59,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/ba95d38932a840a2a1684537fbedad3e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/ba95d38932a840a2a1684537fbedad3e 2023-04-28 04:55:59,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/ba95d38932a840a2a1684537fbedad3e, entries=100, sequenceid=451, filesize=9.5 K 2023-04-28 04:55:59,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1279ms, sequenceid=451, compaction requested=true 2023-04-28 04:55:59,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:55:59,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:59,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:59,775 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:59,775 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:55:59,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:55:59,776 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:59,776 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/A is initiating minor compaction (all files) 2023-04-28 04:55:59,776 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:55:59,776 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/A in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:59,776 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 32a2cc8961b71f647281a29aaf4ac2d5/B is initiating minor compaction (all files) 2023-04-28 04:55:59,776 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 32a2cc8961b71f647281a29aaf4ac2d5/B in TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:55:59,776 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/916236f2e585448e9507ff2f4092f44d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ddd880ccccc74398933ff78ad2f492b4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9400b783e55f4e93a7591a78bee19819] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=34.2 K 2023-04-28 04:55:59,776 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/da9fb6df05f04648b41b1bdcdcf98eaf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/302bd6bcb78541e9a95e7c0dfa2a7895, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/56e23336cfab4c11b22b054a199e6ae9] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp, totalSize=34.2 K 2023-04-28 04:55:59,776 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 916236f2e585448e9507ff2f4092f44d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682657756678 2023-04-28 04:55:59,777 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting da9fb6df05f04648b41b1bdcdcf98eaf, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682657756678 2023-04-28 04:55:59,777 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ddd880ccccc74398933ff78ad2f492b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1682657756727 2023-04-28 04:55:59,777 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 302bd6bcb78541e9a95e7c0dfa2a7895, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1682657756727 2023-04-28 04:55:59,777 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9400b783e55f4e93a7591a78bee19819, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682657758491 2023-04-28 04:55:59,777 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 56e23336cfab4c11b22b054a199e6ae9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682657758491 2023-04-28 04:55:59,784 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#A#compaction#93 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:55:59,787 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 32a2cc8961b71f647281a29aaf4ac2d5#B#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:00,203 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/21041c8025b2434d91edb8fc17a5e158 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/21041c8025b2434d91edb8fc17a5e158 2023-04-28 04:56:00,207 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/e6ea7bf4882843e0b9c7123e7a417b48 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/e6ea7bf4882843e0b9c7123e7a417b48 2023-04-28 04:56:00,214 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/A of 32a2cc8961b71f647281a29aaf4ac2d5 into 21041c8025b2434d91edb8fc17a5e158(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:00,215 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:56:00,215 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/A, priority=13, startTime=1682657759775; duration=0sec 2023-04-28 04:56:00,215 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:00,215 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 04:56:00,216 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 04:56:00,216 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 32a2cc8961b71f647281a29aaf4ac2d5/B of 32a2cc8961b71f647281a29aaf4ac2d5 into e6ea7bf4882843e0b9c7123e7a417b48(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:00,216 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 04:56:00,216 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:56:00,216 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:56:00,216 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5., storeName=32a2cc8961b71f647281a29aaf4ac2d5/B, priority=13, startTime=1682657759775; duration=0sec 2023-04-28 04:56:00,216 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:00,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:56:00,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-28 04:56:00,724 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6be8c20b to 127.0.0.1:52932 2023-04-28 04:56:00,724 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:00,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657758600, max timestamp=1682657760721Num uniques 1018; 2023-04-28 04:56:00,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 04:56:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:56:00,726 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x046b5e40 to 127.0.0.1:52932 2023-04-28 04:56:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:00,726 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:56:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:56:00,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:00,732 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbd52be to 127.0.0.1:52932 2023-04-28 04:56:00,732 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:00,868 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13e837e0 to 127.0.0.1:52932 2023-04-28 04:56:00,868 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:00,871 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6362e5a0 to 127.0.0.1:52932 2023-04-28 04:56:00,871 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 150 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5629 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5476 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 2381 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 7143 rows 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 2374 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 7116 rows 2023-04-28 04:56:00,872 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:56:00,873 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2337cbeb to 127.0.0.1:52932 2023-04-28 04:56:00,873 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:00,877 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 04:56:00,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$11(2418): Client=jenkins//136.243.104.168 disable TestAcidGuarantees 2023-04-28 04:56:00,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:00,891 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657760891"}]},"ts":"1682657760891"} 2023-04-28 04:56:00,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:00,892 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 04:56:00,900 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 04:56:00,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, UNASSIGN}] 2023-04-28 04:56:00,904 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, UNASSIGN 2023-04-28 04:56:00,905 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=32a2cc8961b71f647281a29aaf4ac2d5, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:00,905 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657760905"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657760905"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657760905"}]},"ts":"1682657760905"} 2023-04-28 04:56:00,907 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:56:00,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:01,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:56:01,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 32a2cc8961b71f647281a29aaf4ac2d5, disabling compactions & flushes 2023-04-28 04:56:01,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:56:01,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/a6f9d3229e5f47b5a101bcee65b98f96 2023-04-28 04:56:01,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/509019801a60412d8f25b9d1f7cfec57 2023-04-28 04:56:01,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:01,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/c6c132bdb515492796b5c313f018b452 2023-04-28 04:56:01,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/a6f9d3229e5f47b5a101bcee65b98f96 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a6f9d3229e5f47b5a101bcee65b98f96 2023-04-28 04:56:01,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a6f9d3229e5f47b5a101bcee65b98f96, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 04:56:01,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/509019801a60412d8f25b9d1f7cfec57 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509019801a60412d8f25b9d1f7cfec57 2023-04-28 04:56:01,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509019801a60412d8f25b9d1f7cfec57, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 04:56:01,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/c6c132bdb515492796b5c313f018b452 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c6c132bdb515492796b5c313f018b452 2023-04-28 04:56:01,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c6c132bdb515492796b5c313f018b452, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 04:56:01,605 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~109.58 KB/112210, heapSize ~312.08 KB/319568, currentSize=20.13 KB/20610 for 32a2cc8961b71f647281a29aaf4ac2d5 in 882ms, sequenceid=495, compaction requested=true 2023-04-28 04:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:56:01,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:56:01,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:01,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. after waiting 0 ms 2023-04-28 04:56:01,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:56:01,605 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:56:01,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing 32a2cc8961b71f647281a29aaf4ac2d5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 04:56:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:01,605 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:56:01,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=A 2023-04-28 04:56:01,605 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. because compaction request was cancelled 2023-04-28 04:56:01,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:01,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=B 2023-04-28 04:56:01,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:01,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 32a2cc8961b71f647281a29aaf4ac2d5, store=C 2023-04-28 04:56:01,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:01,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:02,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/81536b1a9407426b803b1cb501b2cac6 2023-04-28 04:56:02,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/509301f2b09b4f098865a25e79aa250c 2023-04-28 04:56:02,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/5195cf6f023740c9be00292ba3162911 2023-04-28 04:56:02,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/A/81536b1a9407426b803b1cb501b2cac6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/81536b1a9407426b803b1cb501b2cac6 2023-04-28 04:56:02,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/81536b1a9407426b803b1cb501b2cac6, entries=100, sequenceid=501, filesize=9.5 K 2023-04-28 04:56:02,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/B/509301f2b09b4f098865a25e79aa250c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509301f2b09b4f098865a25e79aa250c 2023-04-28 04:56:02,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509301f2b09b4f098865a25e79aa250c, entries=100, sequenceid=501, filesize=9.5 K 2023-04-28 04:56:02,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/.tmp/C/5195cf6f023740c9be00292ba3162911 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/5195cf6f023740c9be00292ba3162911 2023-04-28 04:56:02,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/5195cf6f023740c9be00292ba3162911, entries=100, sequenceid=501, filesize=9.5 K 2023-04-28 04:56:02,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 32a2cc8961b71f647281a29aaf4ac2d5 in 1257ms, sequenceid=501, compaction requested=true 2023-04-28 04:56:02,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/95f4783a63e6409ca27d1e57969903ed, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9f15d1e8d382462182670d60bfc0b6f7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9e846719d784474eba03d5e920ba4e1e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/f5f96d73a51a421987183d7d8470e988, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/31087e5c455f4c309ff0cc1e99b15463, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4f7bfa1307594a5783de97263117f0e6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ed4d892f003c46eabbe27416c8677b4a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6939e406ea7444288f2f34713a2ba1fc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/c80020c9bc264eb488950297c0318f5b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/1d0398bf946c4b96880cace5609ae404, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0feae721e6b84febbe34c0eff33e54cb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/51becd6317ee46008b945fa1a264bb22, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6805821b97854364a2e179bab7c95d15, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5844cace2841400e8cd4ba8fab7e7cba, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/92b098c55b9b4c1ab2bfab37c8a88eee, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/828f3b051d7946fd85dc8fe01f2bcd23, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0204fa0edd1d497a88a811af1de0fdc3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/97e9f5f924b54ddcacbd528e782ea6a4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5b34bcbd414e4105baeceb46ebcff119, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/7975595e11194b3486c5332164d6d495, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4a643f933af1460682962f4efa0b2afe, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a8853160416546ecb59402d717a242a2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/48ee65c3365743f5b33eabfe3cd72c55, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/508fd7bd6f604dfab706a6d2311dac2e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/18075098efb14e86b67175411b466db0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/805ce57e9c264de7b5b6af46dba9b1a4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/916236f2e585448e9507ff2f4092f44d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ccfde3d8ba56439d9f0a84b33157e4fe, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ddd880ccccc74398933ff78ad2f492b4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9400b783e55f4e93a7591a78bee19819] to archive 2023-04-28 04:56:02,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:56:02,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/95f4783a63e6409ca27d1e57969903ed to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/95f4783a63e6409ca27d1e57969903ed 2023-04-28 04:56:02,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9f15d1e8d382462182670d60bfc0b6f7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9f15d1e8d382462182670d60bfc0b6f7 2023-04-28 04:56:02,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9e846719d784474eba03d5e920ba4e1e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9e846719d784474eba03d5e920ba4e1e 2023-04-28 04:56:02,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/3d0b03c3cf6e46c7b4d0bb9f8a5e37ce 2023-04-28 04:56:02,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/f5f96d73a51a421987183d7d8470e988 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/f5f96d73a51a421987183d7d8470e988 2023-04-28 04:56:02,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/31087e5c455f4c309ff0cc1e99b15463 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/31087e5c455f4c309ff0cc1e99b15463 2023-04-28 04:56:02,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4f7bfa1307594a5783de97263117f0e6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4f7bfa1307594a5783de97263117f0e6 2023-04-28 04:56:02,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ed4d892f003c46eabbe27416c8677b4a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ed4d892f003c46eabbe27416c8677b4a 2023-04-28 04:56:02,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6939e406ea7444288f2f34713a2ba1fc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6939e406ea7444288f2f34713a2ba1fc 2023-04-28 04:56:02,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/c80020c9bc264eb488950297c0318f5b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/c80020c9bc264eb488950297c0318f5b 2023-04-28 04:56:02,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/1d0398bf946c4b96880cace5609ae404 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/1d0398bf946c4b96880cace5609ae404 2023-04-28 04:56:02,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0feae721e6b84febbe34c0eff33e54cb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0feae721e6b84febbe34c0eff33e54cb 2023-04-28 04:56:02,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/51becd6317ee46008b945fa1a264bb22 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/51becd6317ee46008b945fa1a264bb22 2023-04-28 04:56:02,914 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6805821b97854364a2e179bab7c95d15 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/6805821b97854364a2e179bab7c95d15 2023-04-28 04:56:02,916 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5844cace2841400e8cd4ba8fab7e7cba to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5844cace2841400e8cd4ba8fab7e7cba 2023-04-28 04:56:02,918 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/92b098c55b9b4c1ab2bfab37c8a88eee to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/92b098c55b9b4c1ab2bfab37c8a88eee 2023-04-28 04:56:02,919 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/828f3b051d7946fd85dc8fe01f2bcd23 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/828f3b051d7946fd85dc8fe01f2bcd23 2023-04-28 04:56:02,920 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0204fa0edd1d497a88a811af1de0fdc3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/0204fa0edd1d497a88a811af1de0fdc3 2023-04-28 04:56:02,921 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/97e9f5f924b54ddcacbd528e782ea6a4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/97e9f5f924b54ddcacbd528e782ea6a4 2023-04-28 04:56:02,922 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5b34bcbd414e4105baeceb46ebcff119 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/5b34bcbd414e4105baeceb46ebcff119 2023-04-28 04:56:02,923 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/7975595e11194b3486c5332164d6d495 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/7975595e11194b3486c5332164d6d495 2023-04-28 04:56:02,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4a643f933af1460682962f4efa0b2afe to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/4a643f933af1460682962f4efa0b2afe 2023-04-28 04:56:02,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a8853160416546ecb59402d717a242a2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a8853160416546ecb59402d717a242a2 2023-04-28 04:56:02,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/48ee65c3365743f5b33eabfe3cd72c55 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/48ee65c3365743f5b33eabfe3cd72c55 2023-04-28 04:56:02,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/508fd7bd6f604dfab706a6d2311dac2e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/508fd7bd6f604dfab706a6d2311dac2e 2023-04-28 04:56:02,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/18075098efb14e86b67175411b466db0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/18075098efb14e86b67175411b466db0 2023-04-28 04:56:02,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/805ce57e9c264de7b5b6af46dba9b1a4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/805ce57e9c264de7b5b6af46dba9b1a4 2023-04-28 04:56:02,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/916236f2e585448e9507ff2f4092f44d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/916236f2e585448e9507ff2f4092f44d 2023-04-28 04:56:02,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ccfde3d8ba56439d9f0a84b33157e4fe to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ccfde3d8ba56439d9f0a84b33157e4fe 2023-04-28 04:56:02,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ddd880ccccc74398933ff78ad2f492b4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/ddd880ccccc74398933ff78ad2f492b4 2023-04-28 04:56:02,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9400b783e55f4e93a7591a78bee19819 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/9400b783e55f4e93a7591a78bee19819 2023-04-28 04:56:02,953 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/d50c18c6024a4d0e8735f206e42092f1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ad3ec3b951f1421f8d52ac09c68ba3b4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/69840abb92dc478f9895e160efbdcb66, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8246c8840f074a538b02eeb26e6b4645, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8088ec9c0ee24ecdbea98c104b52d5a9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c538e8efeecc4e41940d81a65d327999, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/014efb7e11df436d8d32673321257baf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/b3fbc0a037cb4a0794dc733b7b4944c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/2aad44304636435d92c6c433a77a6c4b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/577e44d761014885affaab623ce31f6c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c44c4dbfcd8c485f8ca7d416fc55515f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/400fbb6acdc74c0580ab9d34fc20a6ab, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dba94690c54d46b48b690214e9e442f7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ea3a07a3a7a1439090b92bbf111f82d8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/104350af7f344724977503f0a3d4160b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/54450447cfc64f48b7d652cbb5c8612e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/073d27c3c7cc47da9c86bf7d0fd3c38b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/daeb37dde7e546f8b97fb6aa97e84b44, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/48691b4ef0564b82a0777e80704230e5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/9f6c5af6bad145d59af54cacb0745888, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/46b8679e96bd4abb982d56006192c331, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c9dab642941d4602be0638dd355fd961, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/cb09787fbc95495aa11b635abe6a4127, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c171551d939147f8ae25b1a7f7f946f0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dd5a9e2032164dca8fd0524f78b84ba4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c4f4dd8e481b42fca469fe13eaa415ca, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/a6c604c6174946c3b49c6b00f490fd04, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/da9fb6df05f04648b41b1bdcdcf98eaf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8ed2f3f70ec34e61995bef9286f3fc87, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/302bd6bcb78541e9a95e7c0dfa2a7895, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/56e23336cfab4c11b22b054a199e6ae9] to archive 2023-04-28 04:56:02,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:56:02,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/d50c18c6024a4d0e8735f206e42092f1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/d50c18c6024a4d0e8735f206e42092f1 2023-04-28 04:56:02,956 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ad3ec3b951f1421f8d52ac09c68ba3b4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ad3ec3b951f1421f8d52ac09c68ba3b4 2023-04-28 04:56:02,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/69840abb92dc478f9895e160efbdcb66 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/69840abb92dc478f9895e160efbdcb66 2023-04-28 04:56:02,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8246c8840f074a538b02eeb26e6b4645 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8246c8840f074a538b02eeb26e6b4645 2023-04-28 04:56:02,959 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8088ec9c0ee24ecdbea98c104b52d5a9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8088ec9c0ee24ecdbea98c104b52d5a9 2023-04-28 04:56:02,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c538e8efeecc4e41940d81a65d327999 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c538e8efeecc4e41940d81a65d327999 2023-04-28 04:56:02,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/014efb7e11df436d8d32673321257baf to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/014efb7e11df436d8d32673321257baf 2023-04-28 04:56:02,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/b3fbc0a037cb4a0794dc733b7b4944c0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/b3fbc0a037cb4a0794dc733b7b4944c0 2023-04-28 04:56:02,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/2aad44304636435d92c6c433a77a6c4b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/2aad44304636435d92c6c433a77a6c4b 2023-04-28 04:56:02,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/577e44d761014885affaab623ce31f6c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/577e44d761014885affaab623ce31f6c 2023-04-28 04:56:02,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c44c4dbfcd8c485f8ca7d416fc55515f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c44c4dbfcd8c485f8ca7d416fc55515f 2023-04-28 04:56:02,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/400fbb6acdc74c0580ab9d34fc20a6ab to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/400fbb6acdc74c0580ab9d34fc20a6ab 2023-04-28 04:56:02,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dba94690c54d46b48b690214e9e442f7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dba94690c54d46b48b690214e9e442f7 2023-04-28 04:56:02,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ea3a07a3a7a1439090b92bbf111f82d8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/ea3a07a3a7a1439090b92bbf111f82d8 2023-04-28 04:56:02,971 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/104350af7f344724977503f0a3d4160b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/104350af7f344724977503f0a3d4160b 2023-04-28 04:56:02,972 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/54450447cfc64f48b7d652cbb5c8612e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/54450447cfc64f48b7d652cbb5c8612e 2023-04-28 04:56:02,973 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/073d27c3c7cc47da9c86bf7d0fd3c38b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/073d27c3c7cc47da9c86bf7d0fd3c38b 2023-04-28 04:56:02,974 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/daeb37dde7e546f8b97fb6aa97e84b44 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/daeb37dde7e546f8b97fb6aa97e84b44 2023-04-28 04:56:02,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/48691b4ef0564b82a0777e80704230e5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/48691b4ef0564b82a0777e80704230e5 2023-04-28 04:56:02,976 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/9f6c5af6bad145d59af54cacb0745888 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/9f6c5af6bad145d59af54cacb0745888 2023-04-28 04:56:02,977 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/46b8679e96bd4abb982d56006192c331 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/46b8679e96bd4abb982d56006192c331 2023-04-28 04:56:02,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c9dab642941d4602be0638dd355fd961 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c9dab642941d4602be0638dd355fd961 2023-04-28 04:56:02,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/cb09787fbc95495aa11b635abe6a4127 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/cb09787fbc95495aa11b635abe6a4127 2023-04-28 04:56:02,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c171551d939147f8ae25b1a7f7f946f0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c171551d939147f8ae25b1a7f7f946f0 2023-04-28 04:56:02,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dd5a9e2032164dca8fd0524f78b84ba4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/dd5a9e2032164dca8fd0524f78b84ba4 2023-04-28 04:56:02,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c4f4dd8e481b42fca469fe13eaa415ca to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/c4f4dd8e481b42fca469fe13eaa415ca 2023-04-28 04:56:02,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/a6c604c6174946c3b49c6b00f490fd04 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/a6c604c6174946c3b49c6b00f490fd04 2023-04-28 04:56:02,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/da9fb6df05f04648b41b1bdcdcf98eaf to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/da9fb6df05f04648b41b1bdcdcf98eaf 2023-04-28 04:56:02,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8ed2f3f70ec34e61995bef9286f3fc87 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/8ed2f3f70ec34e61995bef9286f3fc87 2023-04-28 04:56:02,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/302bd6bcb78541e9a95e7c0dfa2a7895 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/302bd6bcb78541e9a95e7c0dfa2a7895 2023-04-28 04:56:02,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/56e23336cfab4c11b22b054a199e6ae9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/56e23336cfab4c11b22b054a199e6ae9 2023-04-28 04:56:02,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/fb78e05fc31044448358d5fb9ad73e49, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/4b5d7860dad24289ad54910ab3327cb6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/abfd24e509bc46bbb10a34f43683f7dd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/93488f29591c49bbbde79aba01f1d7a5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c272f43486a241e89814e4c93f81c09a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/94a94ac404ba4bda86d7a1e45b28ee8d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d9e9ef1022c44573b523b2af8081c8af, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/a5801c64989f45e3bf927db124ea004b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/343acae1dfb44db89cc4252ae1ef03f1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/248aaf45a18f4952aaae7833db392f28, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/0d1d0238f3ec4083a44f3788cddeebf9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eaaeb6403a548feb5f552d9dc3b8972, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/17220c2da1054e1f98e971685035b490, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f896a95e5cc64eb28ca0ba4e958161db, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/32471823ccc8425eac085d8dcfdd11a7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d48e3453d31247e7a0084febecc19cad, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/99c45d324b6f45d986fb1e32c9c9f210, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/bc1ecab3f12e486caa5bf88843e7fe6f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3e3c3975554340b29910f11d1c0e87ed, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eff271131d94c45ad6c7fbf0a564d2f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/881c8e47a43b4c41b62083b0223ddd8e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f67565650e9d4f34a22b0e3f6291b88d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8e38295e0de74e5b98c36a2a69ebb135, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c25c5bc0e6174b1ea256568925fdff7e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d2c65d28332c49c999a4975dcb21850d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/69b76e81aef44f708ab2463341ac77ef, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/62aa041d699744768e2b8818f4691fea, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/9a261064c551443ca2ee849f1043f81c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/cda4457585a346418151905041a70101] to archive 2023-04-28 04:56:02,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:56:02,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/fb78e05fc31044448358d5fb9ad73e49 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/fb78e05fc31044448358d5fb9ad73e49 2023-04-28 04:56:02,999 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/4b5d7860dad24289ad54910ab3327cb6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/4b5d7860dad24289ad54910ab3327cb6 2023-04-28 04:56:02,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:03,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/abfd24e509bc46bbb10a34f43683f7dd to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/abfd24e509bc46bbb10a34f43683f7dd 2023-04-28 04:56:03,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/93488f29591c49bbbde79aba01f1d7a5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/93488f29591c49bbbde79aba01f1d7a5 2023-04-28 04:56:03,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c272f43486a241e89814e4c93f81c09a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c272f43486a241e89814e4c93f81c09a 2023-04-28 04:56:03,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/94a94ac404ba4bda86d7a1e45b28ee8d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/94a94ac404ba4bda86d7a1e45b28ee8d 2023-04-28 04:56:03,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d9e9ef1022c44573b523b2af8081c8af to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d9e9ef1022c44573b523b2af8081c8af 2023-04-28 04:56:03,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/a5801c64989f45e3bf927db124ea004b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/a5801c64989f45e3bf927db124ea004b 2023-04-28 04:56:03,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/343acae1dfb44db89cc4252ae1ef03f1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/343acae1dfb44db89cc4252ae1ef03f1 2023-04-28 04:56:03,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/248aaf45a18f4952aaae7833db392f28 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/248aaf45a18f4952aaae7833db392f28 2023-04-28 04:56:03,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/0d1d0238f3ec4083a44f3788cddeebf9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/0d1d0238f3ec4083a44f3788cddeebf9 2023-04-28 04:56:03,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eaaeb6403a548feb5f552d9dc3b8972 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eaaeb6403a548feb5f552d9dc3b8972 2023-04-28 04:56:03,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/17220c2da1054e1f98e971685035b490 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/17220c2da1054e1f98e971685035b490 2023-04-28 04:56:03,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f896a95e5cc64eb28ca0ba4e958161db to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f896a95e5cc64eb28ca0ba4e958161db 2023-04-28 04:56:03,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/32471823ccc8425eac085d8dcfdd11a7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/32471823ccc8425eac085d8dcfdd11a7 2023-04-28 04:56:03,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d48e3453d31247e7a0084febecc19cad to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d48e3453d31247e7a0084febecc19cad 2023-04-28 04:56:03,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/99c45d324b6f45d986fb1e32c9c9f210 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/99c45d324b6f45d986fb1e32c9c9f210 2023-04-28 04:56:03,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/bc1ecab3f12e486caa5bf88843e7fe6f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/bc1ecab3f12e486caa5bf88843e7fe6f 2023-04-28 04:56:03,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3e3c3975554340b29910f11d1c0e87ed to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3e3c3975554340b29910f11d1c0e87ed 2023-04-28 04:56:03,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eff271131d94c45ad6c7fbf0a564d2f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8eff271131d94c45ad6c7fbf0a564d2f 2023-04-28 04:56:03,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/881c8e47a43b4c41b62083b0223ddd8e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/881c8e47a43b4c41b62083b0223ddd8e 2023-04-28 04:56:03,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f67565650e9d4f34a22b0e3f6291b88d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/f67565650e9d4f34a22b0e3f6291b88d 2023-04-28 04:56:03,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8e38295e0de74e5b98c36a2a69ebb135 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/8e38295e0de74e5b98c36a2a69ebb135 2023-04-28 04:56:03,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c25c5bc0e6174b1ea256568925fdff7e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c25c5bc0e6174b1ea256568925fdff7e 2023-04-28 04:56:03,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d2c65d28332c49c999a4975dcb21850d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/d2c65d28332c49c999a4975dcb21850d 2023-04-28 04:56:03,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/69b76e81aef44f708ab2463341ac77ef to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/69b76e81aef44f708ab2463341ac77ef 2023-04-28 04:56:03,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/62aa041d699744768e2b8818f4691fea to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/62aa041d699744768e2b8818f4691fea 2023-04-28 04:56:03,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/9a261064c551443ca2ee849f1043f81c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/9a261064c551443ca2ee849f1043f81c 2023-04-28 04:56:03,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/cda4457585a346418151905041a70101 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/cda4457585a346418151905041a70101 2023-04-28 04:56:03,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/recovered.edits/504.seqid, newMaxSeqId=504, maxSeqId=1 2023-04-28 04:56:03,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5. 2023-04-28 04:56:03,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 32a2cc8961b71f647281a29aaf4ac2d5: 2023-04-28 04:56:03,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed 32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:56:03,035 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=32a2cc8961b71f647281a29aaf4ac2d5, regionState=CLOSED 2023-04-28 04:56:03,035 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657763035"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657763035"}]},"ts":"1682657763035"} 2023-04-28 04:56:03,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-28 04:56:03,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 32a2cc8961b71f647281a29aaf4ac2d5, server=jenkins-hbase12.apache.org,37663,1682657733313 in 2.1290 sec 2023-04-28 04:56:03,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-28 04:56:03,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=32a2cc8961b71f647281a29aaf4ac2d5, UNASSIGN in 2.1360 sec 2023-04-28 04:56:03,040 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657763040"}]},"ts":"1682657763040"} 2023-04-28 04:56:03,041 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 04:56:03,047 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 04:56:03,049 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1660 sec 2023-04-28 04:56:05,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 04:56:05,001 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-28 04:56:05,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$5(2228): Client=jenkins//136.243.104.168 delete TestAcidGuarantees 2023-04-28 04:56:05,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,017 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,018 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 04:56:05,022 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:56:05,026 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/recovered.edits] 2023-04-28 04:56:05,030 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/21041c8025b2434d91edb8fc17a5e158 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/21041c8025b2434d91edb8fc17a5e158 2023-04-28 04:56:05,031 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/81536b1a9407426b803b1cb501b2cac6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/81536b1a9407426b803b1cb501b2cac6 2023-04-28 04:56:05,033 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a6f9d3229e5f47b5a101bcee65b98f96 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/A/a6f9d3229e5f47b5a101bcee65b98f96 2023-04-28 04:56:05,035 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509019801a60412d8f25b9d1f7cfec57 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509019801a60412d8f25b9d1f7cfec57 2023-04-28 04:56:05,037 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509301f2b09b4f098865a25e79aa250c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/509301f2b09b4f098865a25e79aa250c 2023-04-28 04:56:05,038 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/e6ea7bf4882843e0b9c7123e7a417b48 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/B/e6ea7bf4882843e0b9c7123e7a417b48 2023-04-28 04:56:05,040 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3a3dec2ffe644820a12d67eeb9017a9c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/3a3dec2ffe644820a12d67eeb9017a9c 2023-04-28 04:56:05,042 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/5195cf6f023740c9be00292ba3162911 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/5195cf6f023740c9be00292ba3162911 2023-04-28 04:56:05,043 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/ba95d38932a840a2a1684537fbedad3e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/ba95d38932a840a2a1684537fbedad3e 2023-04-28 04:56:05,044 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c6c132bdb515492796b5c313f018b452 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/C/c6c132bdb515492796b5c313f018b452 2023-04-28 04:56:05,047 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/recovered.edits/504.seqid to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5/recovered.edits/504.seqid 2023-04-28 04:56:05,048 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/32a2cc8961b71f647281a29aaf4ac2d5 2023-04-28 04:56:05,048 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:56:05,050 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,068 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 04:56:05,104 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 04:56:05,106 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,106 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 04:56:05,106 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682657765106"}]},"ts":"9223372036854775807"} 2023-04-28 04:56:05,108 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 04:56:05,108 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 32a2cc8961b71f647281a29aaf4ac2d5, NAME => 'TestAcidGuarantees,,1682657737350.32a2cc8961b71f647281a29aaf4ac2d5.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 04:56:05,108 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 04:56:05,108 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682657765108"}]},"ts":"9223372036854775807"} 2023-04-28 04:56:05,110 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 04:56:05,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 04:56:05,122 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 113 msec 2023-04-28 04:56:05,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 04:56:05,323 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-28 04:56:05,346 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=233 (was 206) Potentially hanging thread: member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase12:37663-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x76732ff5-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_224659685_15 at /127.0.0.1:47028 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x76732ff5-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_224659685_15 at /127.0.0.1:46976 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x76732ff5-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x76732ff5-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=548 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=301 (was 211) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=9950 (was 10913) 2023-04-28 04:56:05,363 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233, OpenFileDescriptor=548, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=171, AvailableMemoryMB=9949 2023-04-28 04:56:05,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:56:05,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$4(2112): Client=jenkins//136.243.104.168 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:56:05,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:05,368 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:56:05,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(697): Client=jenkins//136.243.104.168 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-28 04:56:05,369 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:56:05,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 04:56:05,371 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:05,371 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 empty. 2023-04-28 04:56:05,371 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:05,372 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:56:05,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 04:56:05,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 04:56:05,781 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:56:05,787 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 17eb93b75efa028513e70ba389775739, NAME => 'TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:56:05,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 04:56:06,195 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:56:06,195 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 17eb93b75efa028513e70ba389775739, disabling compactions & flushes 2023-04-28 04:56:06,195 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,195 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,195 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. after waiting 0 ms 2023-04-28 04:56:06,195 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,195 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,195 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:06,200 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:56:06,203 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657766202"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657766202"}]},"ts":"1682657766202"} 2023-04-28 04:56:06,205 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:56:06,206 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:56:06,206 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657766206"}]},"ts":"1682657766206"} 2023-04-28 04:56:06,207 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 04:56:06,269 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, ASSIGN}] 2023-04-28 04:56:06,272 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, ASSIGN 2023-04-28 04:56:06,274 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:56:06,426 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:06,426 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657766425"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657766425"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657766425"}]},"ts":"1682657766425"} 2023-04-28 04:56:06,429 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:56:06,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 04:56:06,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 17eb93b75efa028513e70ba389775739, NAME => 'TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:56:06,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:56:06,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,590 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,592 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A 2023-04-28 04:56:06,592 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A 2023-04-28 04:56:06,592 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:06,592 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17eb93b75efa028513e70ba389775739 columnFamilyName A 2023-04-28 04:56:06,593 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(310): Store=17eb93b75efa028513e70ba389775739/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:06,593 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,595 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B 2023-04-28 04:56:06,595 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B 2023-04-28 04:56:06,595 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:06,595 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17eb93b75efa028513e70ba389775739 columnFamilyName B 2023-04-28 04:56:06,596 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(310): Store=17eb93b75efa028513e70ba389775739/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:06,596 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,597 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C 2023-04-28 04:56:06,597 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C 2023-04-28 04:56:06,597 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:06,597 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17eb93b75efa028513e70ba389775739 columnFamilyName C 2023-04-28 04:56:06,598 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(310): Store=17eb93b75efa028513e70ba389775739/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:06,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:56:06,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:06,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:56:06,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 17eb93b75efa028513e70ba389775739; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59333349, jitterRate=-0.11586420238018036}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:56:06,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:06,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., pid=18, masterSystemTime=1682657766581 2023-04-28 04:56:06,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:06,607 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:06,607 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657766607"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657766607"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657766607"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657766607"}]},"ts":"1682657766607"} 2023-04-28 04:56:06,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-28 04:56:06,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 in 180 msec 2023-04-28 04:56:06,612 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-28 04:56:06,612 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, ASSIGN in 341 msec 2023-04-28 04:56:06,613 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:56:06,613 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657766613"}]},"ts":"1682657766613"} 2023-04-28 04:56:06,614 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 04:56:06,638 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:56:06,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2720 sec 2023-04-28 04:56:07,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 04:56:07,476 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-28 04:56:07,477 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x671fd08e to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:07,490 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4894c837, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:07,492 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:07,496 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:07,498 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:56:07,502 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:35818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:56:07,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:56:07,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$12(2454): Client=jenkins//136.243.104.168 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:56:07,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:07,936 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 04:56:07,938 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:56:07,938 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 04:56:07,947 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 04:56:07,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, REOPEN/MOVE}] 2023-04-28 04:56:07,960 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, REOPEN/MOVE 2023-04-28 04:56:07,961 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:07,961 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657767961"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657767961"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657767961"}]},"ts":"1682657767961"} 2023-04-28 04:56:07,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:56:08,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 17eb93b75efa028513e70ba389775739, disabling compactions & flushes 2023-04-28 04:56:08,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. after waiting 0 ms 2023-04-28 04:56:08,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 04:56:08,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:08,122 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 17eb93b75efa028513e70ba389775739 to self. 2023-04-28 04:56:08,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,124 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=CLOSED 2023-04-28 04:56:08,125 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657768124"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657768124"}]},"ts":"1682657768124"} 2023-04-28 04:56:08,129 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-28 04:56:08,129 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 in 163 msec 2023-04-28 04:56:08,129 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=true 2023-04-28 04:56:08,280 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,281 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657768280"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657768280"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657768280"}]},"ts":"1682657768280"} 2023-04-28 04:56:08,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:56:08,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 17eb93b75efa028513e70ba389775739, NAME => 'TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:56:08,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:56:08,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,449 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,450 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A 2023-04-28 04:56:08,450 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A 2023-04-28 04:56:08,450 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:08,453 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17eb93b75efa028513e70ba389775739 columnFamilyName A 2023-04-28 04:56:08,455 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(310): Store=17eb93b75efa028513e70ba389775739/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:08,455 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,457 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B 2023-04-28 04:56:08,457 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B 2023-04-28 04:56:08,457 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:08,457 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17eb93b75efa028513e70ba389775739 columnFamilyName B 2023-04-28 04:56:08,458 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(310): Store=17eb93b75efa028513e70ba389775739/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:08,458 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,459 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C 2023-04-28 04:56:08,459 DEBUG [StoreOpener-17eb93b75efa028513e70ba389775739-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C 2023-04-28 04:56:08,459 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:08,460 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17eb93b75efa028513e70ba389775739 columnFamilyName C 2023-04-28 04:56:08,460 INFO [StoreOpener-17eb93b75efa028513e70ba389775739-1] regionserver.HStore(310): Store=17eb93b75efa028513e70ba389775739/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:08,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:56:08,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 17eb93b75efa028513e70ba389775739; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62298202, jitterRate=-0.07168444991111755}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:56:08,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:08,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., pid=23, masterSystemTime=1682657768437 2023-04-28 04:56:08,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,468 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,469 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657768468"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657768468"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657768468"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657768468"}]},"ts":"1682657768468"} 2023-04-28 04:56:08,472 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-28 04:56:08,472 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 in 186 msec 2023-04-28 04:56:08,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-28 04:56:08,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, REOPEN/MOVE in 513 msec 2023-04-28 04:56:08,477 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-28 04:56:08,477 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 530 msec 2023-04-28 04:56:08,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 964 msec 2023-04-28 04:56:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-28 04:56:08,488 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x2d781def to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,543 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19b40939, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,543 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x7acc1387 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,554 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@633c81ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,554 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x7be739cc to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,558 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:56:08,563 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61ad5d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,564 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x486fba63 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,574 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43a14bcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,575 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x50feca84 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,584 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6577ce5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,585 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x2d5bdceb to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,595 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7605c204, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,596 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x651f745a to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,606 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16505096, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,606 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x1d429358 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,616 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e1990e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,616 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x3432fa3f to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:08,627 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d6367b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:08,640 DEBUG [hconnection-0x7fbb99f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,641 DEBUG [hconnection-0xaf12574-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:08,642 DEBUG [hconnection-0x5a7ed255-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,642 DEBUG [hconnection-0x1be1582a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,644 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52010, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,646 DEBUG [hconnection-0x6723eb07-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,646 DEBUG [hconnection-0x6e559c63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,646 DEBUG [hconnection-0x1eaffc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,652 DEBUG [hconnection-0x170c6624-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,652 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,652 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,652 DEBUG [hconnection-0x3d4ffe84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:08,653 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,654 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,659 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,661 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52058, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,668 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,668 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:52060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:08,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:08,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:08,672 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:08,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:08,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:08,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:08,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,679 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:08,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:56:08,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:08,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:08,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,691 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:08,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:08,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:08,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:08,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:08,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:08,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:08,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:08,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:08,693 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:08,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:08,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:08,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:08,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:08,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:08,702 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,702 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:08,702 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:08,702 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:08,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:08,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:08,703 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:08,710 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,710 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:08,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7407e3e2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:08,712 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:08,712 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,721 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,721 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:08,721 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:08,721 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:08,723 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,723 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682657768704, max timestamp=1682657768724Num uniques 859; 2023-04-28 04:56:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682657768704, max timestamp=1682657768724Num uniques 859; 2023-04-28 04:56:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:08,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657768704, max timestamp=1682657768724Num uniques 859; 2023-04-28 04:56:08,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:08,731 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:08,731 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,731 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:08,731 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:08,736 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:08,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:08,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:08,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:08,736 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:08,739 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281b6b6c963152452995f232f0b9ed030e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281b6b6c963152452995f232f0b9ed030e 2023-04-28 04:56:08,742 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,742 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:08,742 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:08,743 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,744 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657828745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657828746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657828749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657828749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657828751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,752 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:08,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:08,752 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,752 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:08,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:08,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:08,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:08,752 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,752 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:08,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:08,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,753 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:08,753 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:08,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:08,772 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/5a265b5341e6488d9360e0871ea050dd 2023-04-28 04:56:08,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/76a3137838da4416b604c88444174e59 2023-04-28 04:56:08,853 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:08,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:08,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657828853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657828853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657828853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657828853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657828854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:08,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:08,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:08,867 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:08,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:08,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:08,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:08,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,868 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,878 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,878 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:08,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:08,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,879 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:08,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:08,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:08,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:08,884 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:08,884 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:08,884 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:08,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:08,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:08,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:08,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:08,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:08,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:08,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:08,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,893 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:08,893 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:08,899 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,900 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:08,900 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,900 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f7b1419[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:08,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:08,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,910 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:08,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:08,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:08,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:08,921 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:08,921 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,921 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:08,921 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:08,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:08,923 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:08,923 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:08,923 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:08,923 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:08,923 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:08,931 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,931 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:08,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,932 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:08,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:08,935 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:08,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:08,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:08,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:08,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:08,942 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:08,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:08,943 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:08,943 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:08,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:08,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,043 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:09,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:09,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:09,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:09,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:09,060 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:09,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:09,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:09,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:09,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657829056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,061 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,061 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657829056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657829058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657829059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657829065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,125 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:09,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:09,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:09,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:09,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:09,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:09,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:09,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:09,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:09,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:09,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:09,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:09,129 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:09,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:09,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:09,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:09,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:09,168 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:09,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a032f5b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:09,172 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:09,172 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,194 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:09,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:09,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:09,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:09,238 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/abb2e76894f94e5fb5ed116b1852e1fc 2023-04-28 04:56:09,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/5a265b5341e6488d9360e0871ea050dd as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5a265b5341e6488d9360e0871ea050dd 2023-04-28 04:56:09,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5a265b5341e6488d9360e0871ea050dd, entries=150, sequenceid=17, filesize=25.1 K 2023-04-28 04:56:09,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/76a3137838da4416b604c88444174e59 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/76a3137838da4416b604c88444174e59 2023-04-28 04:56:09,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/76a3137838da4416b604c88444174e59, entries=150, sequenceid=17, filesize=11.6 K 2023-04-28 04:56:09,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/abb2e76894f94e5fb5ed116b1852e1fc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/abb2e76894f94e5fb5ed116b1852e1fc 2023-04-28 04:56:09,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/abb2e76894f94e5fb5ed116b1852e1fc, entries=150, sequenceid=17, filesize=11.6 K 2023-04-28 04:56:09,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 17eb93b75efa028513e70ba389775739 in 597ms, sequenceid=17, compaction requested=false 2023-04-28 04:56:09,286 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 04:56:09,286 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:09,286 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,286 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,286 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,287 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:09,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:09,290 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:09,290 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:09,290 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:09,290 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:09,290 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:09,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:09,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-28 04:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:09,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:09,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657829377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657829377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,384 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,384 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,384 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:09,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:09,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657829381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,394 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657829381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657829382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,472 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d1eef46ef1e84ef88c13627bde72a1a0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d1eef46ef1e84ef88c13627bde72a1a0 2023-04-28 04:56:09,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657829482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657829483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657829492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657829500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657829500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:09,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,521 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:09,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,522 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:09,522 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:09,622 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:09,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:09,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:09,625 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:09,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:09,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:09,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:09,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,636 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:09,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:09,637 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,637 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:09,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:09,638 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:09,638 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:09,638 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:09,638 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:09,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:09,647 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,647 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,647 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,647 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:09,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f430540[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:09,648 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,657 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:09,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:09,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:09,658 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,658 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:09,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:09,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:09,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:09,670 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:09,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:09,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:09,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:09,670 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:09,678 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,678 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,679 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:09,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,679 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657829685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657829686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:09,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:09,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:09,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:09,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:09,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:09,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,690 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:09,690 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:09,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:09,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657829694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657829704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657829704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,790 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:09,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:09,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:09,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:09,793 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:09,793 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:09,793 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:09,793 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:09,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,865 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:09,865 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:09,866 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,866 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:09,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:09,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:09,867 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:09,867 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:09,867 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:09,867 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:09,867 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:09,890 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/c9f2ba63cfa0453e9ba9e7d8a52bf071 2023-04-28 04:56:09,910 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,911 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,911 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:09,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490e7b9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:09,913 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:09,913 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,921 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,921 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:09,921 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,921 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:09,921 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:09,921 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:09,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:09,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:09,931 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:09,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:09,933 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:09,934 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:09,934 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:09,934 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:09,934 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:09,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:09,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:09,942 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,943 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,952 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,952 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:09,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:09,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:09,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:09,952 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,952 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:09,952 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:09,952 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:09,953 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:09,952 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:09,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:09,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:09,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:09,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657829989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657829990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:09,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:09,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657829997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657830007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657830008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,053 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:10,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:10,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:10,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:10,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:10,056 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:10,056 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,056 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:10,056 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:10,056 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,056 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,068 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,068 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,068 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,068 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:10,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:10,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:10,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:10,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:10,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:10,069 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:10,069 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,070 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:10,070 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:10,070 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:10,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:10,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:10,070 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:10,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,106 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,106 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@135cf0ee[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:10,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:10,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,120 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,121 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:10,121 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:10,121 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,121 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:10,121 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:10,131 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:10,131 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,131 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,131 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:10,133 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:10,133 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:10,133 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:10,133 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:10,133 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:10,142 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,142 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,142 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,143 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,145 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:10,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:10,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,153 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:10,153 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,153 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:10,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,253 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:10,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:10,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:10,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:10,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:10,256 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:10,256 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,256 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:10,256 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:10,257 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,257 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,268 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,268 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:10,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:10,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:10,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:10,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:10,270 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,270 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:10,270 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:10,270 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:10,270 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:10,278 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,278 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@366c29[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:10,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:10,280 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,289 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:10,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:10,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:10,289 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,289 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:10,299 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:10,299 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,299 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,300 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:10,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:10,302 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:10,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:10,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:10,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:10,302 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:10,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,311 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,312 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,321 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:10,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:10,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,321 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:10,321 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:10,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/2a52fa26c8104ac1a4c2da8f09805316 2023-04-28 04:56:10,421 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:10,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:10,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:10,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:10,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:10,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:10,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:10,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,436 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,436 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:10,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:10,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:10,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:10,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:10,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:10,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:10,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:10,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:10,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:10,439 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:10,439 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,439 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:10,439 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:10,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:10,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,447 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5708106a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:10,449 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:10,449 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,457 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:10,458 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,458 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:10,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:10,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:10,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:10,470 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:10,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:10,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:10,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:10,470 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:10,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,481 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,489 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:10,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,489 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:10,490 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,490 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:10,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657830495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657830496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657830501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657830513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:10,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657830515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,590 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:10,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:10,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:10,600 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:10,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:10,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:10,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,605 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,605 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,605 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,605 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:10,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:10,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:10,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:10,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:10,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:10,606 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:10,606 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,606 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:10,607 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:10,607 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:10,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:10,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:10,607 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:10,615 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,615 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,616 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,616 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:10,617 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:10,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54a6881d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:10,617 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,626 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:10,626 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:10,626 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:10,626 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:10,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,664 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:10,666 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:10,666 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:10,666 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:10,666 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:10,666 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:10,738 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,738 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,738 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:10,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/388276f526fd44d7bfa64be636bd5366 2023-04-28 04:56:10,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:10,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/c9f2ba63cfa0453e9ba9e7d8a52bf071 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c9f2ba63cfa0453e9ba9e7d8a52bf071 2023-04-28 04:56:10,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c9f2ba63cfa0453e9ba9e7d8a52bf071, entries=150, sequenceid=51, filesize=25.1 K 2023-04-28 04:56:10,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/2a52fa26c8104ac1a4c2da8f09805316 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/2a52fa26c8104ac1a4c2da8f09805316 2023-04-28 04:56:10,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/2a52fa26c8104ac1a4c2da8f09805316, entries=150, sequenceid=51, filesize=11.6 K 2023-04-28 04:56:10,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/388276f526fd44d7bfa64be636bd5366 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/388276f526fd44d7bfa64be636bd5366 2023-04-28 04:56:10,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/388276f526fd44d7bfa64be636bd5366, entries=150, sequenceid=51, filesize=11.6 K 2023-04-28 04:56:10,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 17eb93b75efa028513e70ba389775739 in 1399ms, sequenceid=51, compaction requested=false 2023-04-28 04:56:10,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:10,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:10,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:10,812 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,812 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,812 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:10,812 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:10,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:10,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:10,812 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:10,812 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:10,912 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:10,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:10,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:10,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:10,940 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:10,940 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,940 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:10,940 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:10,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,952 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:10,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:10,952 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:10,953 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:10,953 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:10,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:10,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:10,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:10,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:10,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:10,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:10,956 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:10,956 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:10,956 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:10,956 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:10,956 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 04:56:10,957 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:10,957 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:10,957 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:10,957 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:10,957 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:10,957 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:11,003 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a26f299d790a4d12a1a443a49be99496 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a26f299d790a4d12a1a443a49be99496 2023-04-28 04:56:11,023 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/3280eae95e3b4f1f9745a77a8e56a787 2023-04-28 04:56:11,038 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/a5060ca895c940df9a90bcbd9c8baea4 2023-04-28 04:56:11,056 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/4d5c9db1b1664932b3cfce19fc268fe1 2023-04-28 04:56:11,061 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/3280eae95e3b4f1f9745a77a8e56a787 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3280eae95e3b4f1f9745a77a8e56a787 2023-04-28 04:56:11,066 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3280eae95e3b4f1f9745a77a8e56a787, entries=150, sequenceid=63, filesize=25.1 K 2023-04-28 04:56:11,068 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/a5060ca895c940df9a90bcbd9c8baea4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/a5060ca895c940df9a90bcbd9c8baea4 2023-04-28 04:56:11,074 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/a5060ca895c940df9a90bcbd9c8baea4, entries=150, sequenceid=63, filesize=11.6 K 2023-04-28 04:56:11,075 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/4d5c9db1b1664932b3cfce19fc268fe1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4d5c9db1b1664932b3cfce19fc268fe1 2023-04-28 04:56:11,087 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4d5c9db1b1664932b3cfce19fc268fe1, entries=150, sequenceid=63, filesize=11.6 K 2023-04-28 04:56:11,089 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 17eb93b75efa028513e70ba389775739 in 133ms, sequenceid=63, compaction requested=true 2023-04-28 04:56:11,090 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:11,090 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:11,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:11,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:11,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:11,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,100 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:11,102 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:11,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2009c9e7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:11,102 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,110 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,110 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:11,110 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:11,110 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:11,110 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,110 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:11,120 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,120 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,121 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,121 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:11,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:11,124 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:11,124 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:11,124 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:11,124 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:11,124 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:11,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,163 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:11,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,163 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:11,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,163 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:11,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,263 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:11,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:11,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:11,269 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:11,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:11,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:11,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,278 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,278 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,279 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,279 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:11,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:11,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:11,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:11,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:11,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:11,288 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:11,288 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:11,288 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,288 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:11,288 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:11,288 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:11,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:11,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:11,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:11,299 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,299 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,300 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,300 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:11,301 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:11,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@136daeeb[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:11,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,310 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:11,310 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:11,310 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:11,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:11,320 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:11,321 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,321 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:11,324 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:11,324 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:11,324 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:11,324 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:11,324 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:11,331 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,331 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,331 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,332 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,333 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:11,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:11,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,342 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,342 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:11,342 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,342 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,442 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:11,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:11,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:11,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:11,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:11,456 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:11,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:11,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:11,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:11,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:11,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:11,470 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:11,470 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,470 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:11,470 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:11,470 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:11,470 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,471 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:11,471 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:11,471 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,471 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:11,471 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:11,471 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,471 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:11,471 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:11,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,478 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,479 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,479 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:11,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47177ac[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:11,481 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:11,481 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,489 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,489 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:11,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:11,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:11,499 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:11,499 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,499 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:11,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:11,502 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:11,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:11,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:11,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:11,502 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:11,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,520 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:11,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:11,521 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,521 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,521 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:11,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:56:11,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:11,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:11,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:11,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:11,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:11,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:11,584 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428173bf49a81db40848a13f05c559ef4a2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428173bf49a81db40848a13f05c559ef4a2 2023-04-28 04:56:11,621 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:11,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:11,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:11,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:11,625 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657831630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657831631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657831631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657771598, max timestamp=1682657771634Num uniques 859; 2023-04-28 04:56:11,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657771598, max timestamp=1682657771634Num uniques 859; 2023-04-28 04:56:11,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:11,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:11,636 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657771598, max timestamp=1682657771634Num uniques 859; 2023-04-28 04:56:11,637 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:11,637 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,637 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:11,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:11,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:11,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,638 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:11,640 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,640 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:11,640 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:11,640 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:11,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:11,647 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,647 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,648 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,648 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6477f15f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:11,649 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:11,649 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,657 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:11,658 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,658 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:11,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:11,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:11,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:11,670 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:11,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:11,671 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:11,671 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:11,671 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:11,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657831674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657831677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,681 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,682 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,683 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:11,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:11,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,690 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:11,690 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:11,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:11,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657831737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657831737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657831737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657831778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657831782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,790 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:11,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:11,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:11,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:11,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:11,796 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:11,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:11,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:11,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,805 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:11,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:11,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:11,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:11,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:11,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:11,806 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,807 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:11,807 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:11,807 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:11,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:11,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:11,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:11,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,815 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,816 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,816 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:11,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:11,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@adf5ac7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:11,817 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,826 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,826 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,826 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:11,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:11,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:11,836 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:11,836 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,836 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:11,836 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:11,839 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:11,839 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:11,839 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:11,839 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:11,839 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:11,847 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,847 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,847 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,847 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,848 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,857 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,857 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:11,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:11,857 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,857 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:11,857 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:11,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,858 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:11,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,858 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:11,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,858 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:11,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657831940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657831941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657831942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,958 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:11,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:11,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:11,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:11,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:11,961 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:11,961 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,961 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:11,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:11,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,973 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:11,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:11,973 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,973 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:11,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:11,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:11,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:11,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:11,975 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,975 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:11,975 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:11,975 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:11,975 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:11,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657831981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:11,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:11,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657831984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:11,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:11,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743bb781[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:11,986 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:11,986 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,994 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:11,994 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:11,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:11,994 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:11,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:11,995 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:11,999 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=76, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/8be5cec31be34b808772cbf669f5a07e 2023-04-28 04:56:12,005 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:12,005 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,005 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:12,005 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:12,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:12,007 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:12,007 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:12,007 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:12,007 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:12,007 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:12,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,016 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,017 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,055 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:12,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:12,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:12,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,055 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:12,056 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,056 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:12,056 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,056 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,056 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,156 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:12,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:12,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:12,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:12,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:12,159 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:12,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:12,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:12,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,173 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,173 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,174 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,174 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:12,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:12,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:12,175 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,175 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:12,175 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:12,175 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:12,175 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:12,184 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,184 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,184 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,184 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:12,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7872bd[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:12,190 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:12,190 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,194 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:12,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:12,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:12,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:12,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:12,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,205 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:12,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:12,209 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:12,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:12,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:12,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:12,209 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:12,215 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,215 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,215 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,216 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,217 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,226 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:12,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:12,227 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:12,227 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:12,226 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657832245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657832245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657832245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657832285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657832287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,327 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:12,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:12,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:12,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:12,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:12,330 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:12,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:12,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:12,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,330 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,352 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:12,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:12,354 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,354 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:12,354 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:12,354 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:12,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:12,363 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,363 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,363 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,363 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:12,365 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20de8984[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:12,365 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:12,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:12,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:12,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:12,384 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:12,384 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,384 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,384 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:12,386 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:12,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:12,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:12,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:12,386 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:12,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,395 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,396 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:12,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:12,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,405 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,405 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,405 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:12,405 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:12,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/514b501691ac478187e4c02169d63549 2023-04-28 04:56:12,505 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:12,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:12,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:12,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:12,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:12,508 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:12,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:12,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:12,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,520 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,521 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,521 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:12,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:12,522 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,522 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:12,522 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:12,522 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:12,522 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:12,531 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5027e63a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:12,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:12,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,541 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,542 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,542 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:12,542 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:12,542 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:12,542 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:12,552 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:12,552 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,552 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,552 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:12,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:12,554 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:12,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:12,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:12,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:12,554 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:12,562 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,563 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,563 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,563 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,564 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,603 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,603 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:12,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:12,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,603 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,603 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:12,604 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:12,604 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:12,603 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,704 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:12,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:12,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:12,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:12,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:12,709 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:12,709 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,709 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:12,709 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:12,709 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,709 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,731 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,731 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:12,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:12,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:12,733 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:12,733 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,733 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:12,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:12,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:12,734 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:12,734 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:12,742 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,742 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,742 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,742 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:12,743 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7989c400[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:12,744 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657832751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,753 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657832753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,754 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:12,754 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:12,754 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:12,754 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,754 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:12,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657832754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:12,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:12,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:12,774 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:12,774 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:12,774 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:12,774 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:12,774 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:12,784 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,784 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,784 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:12,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:12,784 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:12,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:12,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:12,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:12,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:12,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,795 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:12,795 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:12,795 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:12,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657832792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:12,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657832794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:12,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/294c5e0c430c45418083119c958afeff 2023-04-28 04:56:12,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/8be5cec31be34b808772cbf669f5a07e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/8be5cec31be34b808772cbf669f5a07e 2023-04-28 04:56:12,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/8be5cec31be34b808772cbf669f5a07e, entries=400, sequenceid=76, filesize=59.0 K 2023-04-28 04:56:12,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/514b501691ac478187e4c02169d63549 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/514b501691ac478187e4c02169d63549 2023-04-28 04:56:12,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/514b501691ac478187e4c02169d63549, entries=150, sequenceid=76, filesize=11.6 K 2023-04-28 04:56:12,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/294c5e0c430c45418083119c958afeff as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/294c5e0c430c45418083119c958afeff 2023-04-28 04:56:12,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/294c5e0c430c45418083119c958afeff, entries=150, sequenceid=76, filesize=11.6 K 2023-04-28 04:56:12,853 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 17eb93b75efa028513e70ba389775739 in 1294ms, sequenceid=76, compaction requested=true 2023-04-28 04:56:12,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:12,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:12,853 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:12,853 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:12,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:12,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:12,854 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 137692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:12,854 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:12,854 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/A is initiating minor compaction (all files) 2023-04-28 04:56:12,854 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/B is initiating minor compaction (all files) 2023-04-28 04:56:12,854 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/A in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,855 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/B in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,855 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/76a3137838da4416b604c88444174e59, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/2a52fa26c8104ac1a4c2da8f09805316, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/a5060ca895c940df9a90bcbd9c8baea4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/514b501691ac478187e4c02169d63549] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=46.4 K 2023-04-28 04:56:12,855 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5a265b5341e6488d9360e0871ea050dd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c9f2ba63cfa0453e9ba9e7d8a52bf071, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3280eae95e3b4f1f9745a77a8e56a787, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/8be5cec31be34b808772cbf669f5a07e] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=134.5 K 2023-04-28 04:56:12,855 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 76a3137838da4416b604c88444174e59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682657768670 2023-04-28 04:56:12,855 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5a265b5341e6488d9360e0871ea050dd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682657768670 2023-04-28 04:56:12,856 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 2a52fa26c8104ac1a4c2da8f09805316, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682657768744 2023-04-28 04:56:12,856 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c9f2ba63cfa0453e9ba9e7d8a52bf071, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682657768744 2023-04-28 04:56:12,857 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3280eae95e3b4f1f9745a77a8e56a787, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682657769374 2023-04-28 04:56:12,857 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a5060ca895c940df9a90bcbd9c8baea4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682657769374 2023-04-28 04:56:12,858 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 8be5cec31be34b808772cbf669f5a07e, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1682657771522 2023-04-28 04:56:12,858 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 514b501691ac478187e4c02169d63549, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1682657771552 2023-04-28 04:56:12,895 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:12,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:12,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:12,901 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#B#compaction#113 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:12,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:12,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:12,902 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:12,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:12,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:12,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,906 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#A#compaction#114 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:12,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,910 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:12,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:12,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:12,911 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:12,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:12,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:12,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:12,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:12,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:12,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:12,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:12,927 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:12,927 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:12,928 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:12,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:12,943 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428766a688d9dda47b1b5109935afd13324 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428766a688d9dda47b1b5109935afd13324 2023-04-28 04:56:13,335 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/d8e895b88c8b4ec18976685573e9701a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d8e895b88c8b4ec18976685573e9701a 2023-04-28 04:56:13,344 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 17eb93b75efa028513e70ba389775739/B of 17eb93b75efa028513e70ba389775739 into d8e895b88c8b4ec18976685573e9701a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:13,344 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:13,344 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/B, priority=12, startTime=1682657772853; duration=0sec 2023-04-28 04:56:13,345 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:13,345 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:13,347 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:13,348 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/C is initiating minor compaction (all files) 2023-04-28 04:56:13,348 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/C in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:13,348 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/abb2e76894f94e5fb5ed116b1852e1fc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/388276f526fd44d7bfa64be636bd5366, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4d5c9db1b1664932b3cfce19fc268fe1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/294c5e0c430c45418083119c958afeff] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=46.4 K 2023-04-28 04:56:13,348 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/5234e3bee59b4bacb2e20048da4a5421 2023-04-28 04:56:13,349 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting abb2e76894f94e5fb5ed116b1852e1fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682657768670 2023-04-28 04:56:13,352 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 388276f526fd44d7bfa64be636bd5366, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682657768744 2023-04-28 04:56:13,353 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4d5c9db1b1664932b3cfce19fc268fe1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682657769374 2023-04-28 04:56:13,353 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 294c5e0c430c45418083119c958afeff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1682657771552 2023-04-28 04:56:13,365 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/e041720ef4ae4bfabf101c41d075ed2d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/e041720ef4ae4bfabf101c41d075ed2d 2023-04-28 04:56:13,372 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 17eb93b75efa028513e70ba389775739/A of 17eb93b75efa028513e70ba389775739 into e041720ef4ae4bfabf101c41d075ed2d(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:13,372 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:13,372 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/A, priority=12, startTime=1682657772853; duration=0sec 2023-04-28 04:56:13,372 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:13,376 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#C#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:13,400 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/3f0164fae78642d9a344fc8e76d092b7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/3f0164fae78642d9a344fc8e76d092b7 2023-04-28 04:56:13,406 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 17eb93b75efa028513e70ba389775739/C of 17eb93b75efa028513e70ba389775739 into 3f0164fae78642d9a344fc8e76d092b7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:13,406 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:13,406 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/C, priority=12, startTime=1682657772853; duration=0sec 2023-04-28 04:56:13,406 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:13,624 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:56:13,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:13,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:13,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657833768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657833770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657833770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,778 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/9810a73debce48bb850abbb6a004b048 2023-04-28 04:56:13,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657833799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657833804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657833871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657833871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:13,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:13,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657833873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657834074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657834074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657834076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,190 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/94ed88a84ad64250b512aeb618052a07 2023-04-28 04:56:14,206 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/5234e3bee59b4bacb2e20048da4a5421 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5234e3bee59b4bacb2e20048da4a5421 2023-04-28 04:56:14,214 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5234e3bee59b4bacb2e20048da4a5421, entries=150, sequenceid=111, filesize=25.1 K 2023-04-28 04:56:14,215 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/9810a73debce48bb850abbb6a004b048 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/9810a73debce48bb850abbb6a004b048 2023-04-28 04:56:14,249 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/9810a73debce48bb850abbb6a004b048, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:56:14,251 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/94ed88a84ad64250b512aeb618052a07 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/94ed88a84ad64250b512aeb618052a07 2023-04-28 04:56:14,256 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/94ed88a84ad64250b512aeb618052a07, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:56:14,257 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 17eb93b75efa028513e70ba389775739 in 1329ms, sequenceid=111, compaction requested=false 2023-04-28 04:56:14,257 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:14,257 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:14,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:14,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,260 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:14,260 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:14,278 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,278 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:14,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41dcac16[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:14,281 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:14,281 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,289 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:14,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:14,289 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:14,299 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:14,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:14,300 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:14,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:14,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:14,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:14,300 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:14,300 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:14,300 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:14,300 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:14,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,310 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:14,311 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,312 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:14,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:14,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:14,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:14,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,321 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:14,321 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:14,321 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:14,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:14,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:14,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:14,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 04:56:14,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:14,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:14,420 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428597722ff9b054577bdcde346f35b11bd to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428597722ff9b054577bdcde346f35b11bd 2023-04-28 04:56:14,421 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:14,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:14,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:14,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657834422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657834424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:14,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:14,428 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:14,428 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:14,428 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:14,428 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:14,429 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,429 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657834426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,437 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:14,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:14,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:14,438 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,438 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:14,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:14,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:14,448 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:14,448 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:14,448 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:14,456 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:14,456 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:14,456 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,457 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:14,457 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:14,458 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:14,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:14,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:14,458 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:14,459 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=126, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/748a4f90d0b2466e9949f51978a837f0 2023-04-28 04:56:14,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,469 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159a9f00[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:14,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:14,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:14,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,479 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:14,479 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:14,479 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:14,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/ad98e35f3c0943f0ba7d43ae663eefff 2023-04-28 04:56:14,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:14,489 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:14,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:14,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:14,492 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:14,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:14,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:14,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:14,492 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:14,499 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,499 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,499 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,501 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,502 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:14,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:14,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,510 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:14,510 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:14,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:14,510 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:14,510 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:14,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:14,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657834528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657834528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657834533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,611 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:14,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:14,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:14,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:14,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:14,614 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:14,614 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:14,614 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:14,614 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:14,615 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,615 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52060 deadline: 1682657834730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657834732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:14,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657834738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,880 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:14,881 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:14,881 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,881 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:14,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:14,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:14,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:14,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:14,882 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:14,882 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:14,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/670085c45ed34f52b7843cca0e9020ef 2023-04-28 04:56:14,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/748a4f90d0b2466e9949f51978a837f0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/748a4f90d0b2466e9949f51978a837f0 2023-04-28 04:56:14,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/748a4f90d0b2466e9949f51978a837f0, entries=200, sequenceid=126, filesize=31.9 K 2023-04-28 04:56:14,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/ad98e35f3c0943f0ba7d43ae663eefff as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/ad98e35f3c0943f0ba7d43ae663eefff 2023-04-28 04:56:14,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/ad98e35f3c0943f0ba7d43ae663eefff, entries=150, sequenceid=126, filesize=11.6 K 2023-04-28 04:56:14,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/670085c45ed34f52b7843cca0e9020ef as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/670085c45ed34f52b7843cca0e9020ef 2023-04-28 04:56:14,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/670085c45ed34f52b7843cca0e9020ef, entries=150, sequenceid=126, filesize=11.6 K 2023-04-28 04:56:14,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 17eb93b75efa028513e70ba389775739 in 549ms, sequenceid=126, compaction requested=true 2023-04-28 04:56:14,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:14,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:14,929 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:14,929 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:14,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:14,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:14,930 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:14,931 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/B is initiating minor compaction (all files) 2023-04-28 04:56:14,931 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/B in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,931 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d8e895b88c8b4ec18976685573e9701a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/9810a73debce48bb850abbb6a004b048, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/ad98e35f3c0943f0ba7d43ae663eefff] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=35.0 K 2023-04-28 04:56:14,931 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:14,931 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/A is initiating minor compaction (all files) 2023-04-28 04:56:14,931 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/A in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,931 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d8e895b88c8b4ec18976685573e9701a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1682657771552 2023-04-28 04:56:14,931 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/e041720ef4ae4bfabf101c41d075ed2d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5234e3bee59b4bacb2e20048da4a5421, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/748a4f90d0b2466e9949f51978a837f0] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=82.3 K 2023-04-28 04:56:14,932 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9810a73debce48bb850abbb6a004b048, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657771661 2023-04-28 04:56:14,932 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting e041720ef4ae4bfabf101c41d075ed2d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1682657771552 2023-04-28 04:56:14,932 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ad98e35f3c0943f0ba7d43ae663eefff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682657773768 2023-04-28 04:56:14,932 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5234e3bee59b4bacb2e20048da4a5421, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657771661 2023-04-28 04:56:14,932 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 748a4f90d0b2466e9949f51978a837f0, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682657773766 2023-04-28 04:56:14,950 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#B#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:14,956 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#A#compaction#123 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:14,964 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/867cf39946654123aa202d227b193f0f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/867cf39946654123aa202d227b193f0f 2023-04-28 04:56:14,970 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/B of 17eb93b75efa028513e70ba389775739 into 867cf39946654123aa202d227b193f0f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:14,970 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:14,970 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/B, priority=13, startTime=1682657774929; duration=0sec 2023-04-28 04:56:14,970 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:14,970 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:14,971 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:14,972 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/C is initiating minor compaction (all files) 2023-04-28 04:56:14,972 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/C in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:14,972 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/3f0164fae78642d9a344fc8e76d092b7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/94ed88a84ad64250b512aeb618052a07, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/670085c45ed34f52b7843cca0e9020ef] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=35.0 K 2023-04-28 04:56:14,972 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 3f0164fae78642d9a344fc8e76d092b7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1682657771552 2023-04-28 04:56:14,972 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 94ed88a84ad64250b512aeb618052a07, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657771661 2023-04-28 04:56:14,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,973 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 670085c45ed34f52b7843cca0e9020ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682657773768 2023-04-28 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,973 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,974 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a63adbf[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:14,974 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:14,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,979 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#C#compaction#124 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:14,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:14,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:14,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:14,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:14,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:14,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:14,994 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:14,994 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:14,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:14,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:14,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:14,997 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:14,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:14,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:14,997 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:14,997 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:15,004 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,005 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,005 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,005 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,006 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/516be6a7aa9a402883e3803dd3301f1a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/516be6a7aa9a402883e3803dd3301f1a 2023-04-28 04:56:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,011 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/C of 17eb93b75efa028513e70ba389775739 into 516be6a7aa9a402883e3803dd3301f1a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:15,011 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:15,011 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/C, priority=13, startTime=1682657774929; duration=0sec 2023-04-28 04:56:15,011 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,015 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:15,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,015 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,016 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,016 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,016 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:15,016 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:15,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:15,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-28 04:56:15,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657774402, max timestamp=1682657775036Num uniques 859; 2023-04-28 04:56:15,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:15,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657774402, max timestamp=1682657775036Num uniques 859; 2023-04-28 04:56:15,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:15,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657774402, max timestamp=1682657775036Num uniques 859; 2023-04-28 04:56:15,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:15,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:15,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:15,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:15,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:15,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:15,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:15,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657775053, max timestamp=1682657775096Num uniques 859; 2023-04-28 04:56:15,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:15,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657775053, max timestamp=1682657775096Num uniques 859; 2023-04-28 04:56:15,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:15,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657775053, max timestamp=1682657775096Num uniques 859; 2023-04-28 04:56:15,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:15,116 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:15,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:15,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:15,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657775111, max timestamp=1682657775128Num uniques 1118; 2023-04-28 04:56:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-28 04:56:15,132 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:15,132 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,132 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:15,132 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657775111, max timestamp=1682657775128Num uniques 1118; 2023-04-28 04:56:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657775111, max timestamp=1682657775128Num uniques 1118; 2023-04-28 04:56:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-28 04:56:15,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-28 04:56:15,133 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,133 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,141 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,142 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,142 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:15,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:15,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:15,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,143 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:15,144 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,144 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:15,144 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:15,144 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:15,144 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:15,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,152 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,152 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,152 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7dcf37[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:15,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:15,154 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,163 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,163 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:15,163 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:15,163 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:15,163 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,163 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:15,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657775152, max timestamp=1682657775169Num uniques 1027; 2023-04-28 04:56:15,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1682657775152, max timestamp=1682657775169Num uniques 1027; 2023-04-28 04:56:15,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=8092, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-28 04:56:15,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8097, totalHeapSize=18648, min timestamp=1682657775152, max timestamp=1682657775169Num uniques 1027; 2023-04-28 04:56:15,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=47027, new segment data size=8097, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-28 04:56:15,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=47023, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-28 04:56:15,173 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,173 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:15,173 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,173 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:15,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:15,175 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:15,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:15,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:15,175 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:15,175 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:15,183 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,183 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,184 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,184 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,187 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,204 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,204 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682657775188, max timestamp=1682657775201Num uniques 1036; 2023-04-28 04:56:15,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682657775188, max timestamp=1682657775201Num uniques 1036; 2023-04-28 04:56:15,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,205 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-28 04:56:15,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:15,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:15,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,205 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:15,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-28 04:56:15,206 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:15,205 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,206 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682657775188, max timestamp=1682657775201Num uniques 1009; 2023-04-28 04:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46223, new segment data size=13096, suffix heap size=132904,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=286 2023-04-28 04:56:15,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:15,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657835258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682657775216, max timestamp=1682657775262Num uniques 1145; 2023-04-28 04:56:15,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-28 04:56:15,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682657775216, max timestamp=1682657775262Num uniques 1145; 2023-04-28 04:56:15,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-28 04:56:15,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682657775216, max timestamp=1682657775262Num uniques 1145; 2023-04-28 04:56:15,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-28 04:56:15,306 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:15,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:15,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657775292, max timestamp=1682657775304Num uniques 1154; 2023-04-28 04:56:15,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-28 04:56:15,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657775292, max timestamp=1682657775304Num uniques 1204; 2023-04-28 04:56:15,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-28 04:56:15,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:15,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657775292, max timestamp=1682657775304Num uniques 1154; 2023-04-28 04:56:15,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-28 04:56:15,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:15,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:15,310 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:15,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:15,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:15,310 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,311 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,320 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,321 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,321 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:15,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:15,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:15,323 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:15,323 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,323 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:15,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:15,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:15,324 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:15,324 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:15,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682657775326, max timestamp=1682657775345Num uniques 1063; 2023-04-28 04:56:15,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 04:56:15,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682657775326, max timestamp=1682657775345Num uniques 1063; 2023-04-28 04:56:15,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-28 04:56:15,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682657775326, max timestamp=1682657775345Num uniques 1013; 2023-04-28 04:56:15,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-28 04:56:15,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10154, totalHeapSize=23328, min timestamp=1682657775376, max timestamp=1682657775383Num uniques 1122; 2023-04-28 04:56:15,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=51374, new segment data size=10154, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-28 04:56:15,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10152, totalHeapSize=23328, min timestamp=1682657775376, max timestamp=1682657775383Num uniques 1122; 2023-04-28 04:56:15,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657835383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10159, totalHeapSize=23328, min timestamp=1682657775376, max timestamp=1682657775383Num uniques 1122; 2023-04-28 04:56:15,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=51379, new segment data size=10159, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-28 04:56:15,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=51372, new segment data size=10152, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-28 04:56:15,412 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,413 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,413 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69657692[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:15,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:15,415 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657775415, max timestamp=1682657775421Num uniques 1081; 2023-04-28 04:56:15,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 04:56:15,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657775415, max timestamp=1682657775421Num uniques 1081; 2023-04-28 04:56:15,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657775415, max timestamp=1682657775421Num uniques 1081; 2023-04-28 04:56:15,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 04:56:15,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 04:56:15,447 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,447 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:15,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:15,447 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:15,447 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:15,454 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ae21111f7e8c447ba674c9bb963ea93d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ae21111f7e8c447ba674c9bb963ea93d 2023-04-28 04:56:15,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682657775434, max timestamp=1682657775452Num uniques 1090; 2023-04-28 04:56:15,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 04:56:15,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682657775434, max timestamp=1682657775452Num uniques 1090; 2023-04-28 04:56:15,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 04:56:15,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682657775434, max timestamp=1682657775452Num uniques 1090; 2023-04-28 04:56:15,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-28 04:56:15,457 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:15,457 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,457 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,457 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:15,460 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:15,460 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:15,460 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:15,460 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:15,460 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:15,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:15,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:15,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,478 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,479 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:15,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,479 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:15,478 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,479 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682657775465, max timestamp=1682657775483Num uniques 1099; 2023-04-28 04:56:15,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-28 04:56:15,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682657775465, max timestamp=1682657775483Num uniques 1099; 2023-04-28 04:56:15,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-28 04:56:15,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682657775465, max timestamp=1682657775483Num uniques 1099; 2023-04-28 04:56:15,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-28 04:56:15,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657835511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657835512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657835514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,579 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:15,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:15,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:15,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:15,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:15,582 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:15,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:15,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:15,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,594 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,594 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,594 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,594 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:15,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:15,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:15,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:15,595 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:15,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:15,596 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,596 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:15,596 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:15,596 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:15,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:15,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:15,596 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:15,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,604 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,605 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,605 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28468621[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:15,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:15,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657835615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657835617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52012 deadline: 1682657835617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,646 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,646 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,646 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:15,646 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:15,646 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:15,646 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:15,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:15,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,668 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:15,670 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:15,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:15,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:15,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:15,670 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:15,678 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,678 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,678 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,679 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:15,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:15,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,689 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,689 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,689 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,787 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/c654e8ecc20e4659a1c98647e463e87a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c654e8ecc20e4659a1c98647e463e87a 2023-04-28 04:56:15,789 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:15,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:15,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:15,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:15,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:15,794 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:15,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:15,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:15,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,796 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/A of 17eb93b75efa028513e70ba389775739 into c654e8ecc20e4659a1c98647e463e87a(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:15,796 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:15,797 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/A, priority=13, startTime=1682657774929; duration=0sec 2023-04-28 04:56:15,797 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:15,804 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:15,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:15,806 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,806 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:15,806 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:15,806 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:15,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:15,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,815 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657835812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,816 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,816 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,818 DEBUG [Thread-809] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657835813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,822 DEBUG [Thread-818] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:15,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657835819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:15,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2fd8c3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:15,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:15,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657835822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52016 deadline: 1682657835825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,836 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,836 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:15,836 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:15,836 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:15,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:15,864 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/a0bf57e4abfb4e9e885b614fb70c761d 2023-04-28 04:56:15,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/359da8f89c2e47e4bb68069c1317ec75 2023-04-28 04:56:15,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/573cc5737ce14e808b1283bfe43e051f 2023-04-28 04:56:15,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/a0bf57e4abfb4e9e885b614fb70c761d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/a0bf57e4abfb4e9e885b614fb70c761d 2023-04-28 04:56:15,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:15,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,917 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:15,921 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:15,921 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:15,921 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:15,921 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:15,921 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:15,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/a0bf57e4abfb4e9e885b614fb70c761d, entries=200, sequenceid=150, filesize=32.1 K 2023-04-28 04:56:15,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/359da8f89c2e47e4bb68069c1317ec75 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/359da8f89c2e47e4bb68069c1317ec75 2023-04-28 04:56:15,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/359da8f89c2e47e4bb68069c1317ec75, entries=150, sequenceid=150, filesize=11.8 K 2023-04-28 04:56:15,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/573cc5737ce14e808b1283bfe43e051f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/573cc5737ce14e808b1283bfe43e051f 2023-04-28 04:56:15,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/573cc5737ce14e808b1283bfe43e051f, entries=150, sequenceid=150, filesize=11.8 K 2023-04-28 04:56:15,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 17eb93b75efa028513e70ba389775739 in 899ms, sequenceid=150, compaction requested=false 2023-04-28 04:56:15,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:15,962 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,962 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:15,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,963 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:15,963 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,964 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:15,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:15,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:15,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:15,973 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:15,973 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:15,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:15,973 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:16,074 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:16,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:16,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:16,077 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:16,077 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:16,077 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:16,077 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:16,077 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:16,077 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:16,089 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:16,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:16,089 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:16,089 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:16,089 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:16,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:16,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:16,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:16,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:16,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:16,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:16,090 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:16,091 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:16,091 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=154.31 KB heapSize=431.63 KB 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:16,091 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:16,125 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c5387b4364734dfe99047c7e9765fa19 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c5387b4364734dfe99047c7e9765fa19 2023-04-28 04:56:16,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:16,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:16,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657836133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657836133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657836137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657836238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657836239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657836239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657836441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657836441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657836441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,535 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/5db41099cbb442b78592e07c610af955 2023-04-28 04:56:16,550 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/d283ef9931a14438baf746b02cd084bf 2023-04-28 04:56:16,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657836745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657836745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657836746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,967 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/b072903ee5d244cda63c9f20366dc0a2 2023-04-28 04:56:16,972 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/5db41099cbb442b78592e07c610af955 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5db41099cbb442b78592e07c610af955 2023-04-28 04:56:16,976 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5db41099cbb442b78592e07c610af955, entries=150, sequenceid=360, filesize=25.4 K 2023-04-28 04:56:16,977 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/d283ef9931a14438baf746b02cd084bf as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d283ef9931a14438baf746b02cd084bf 2023-04-28 04:56:16,980 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d283ef9931a14438baf746b02cd084bf, entries=150, sequenceid=360, filesize=11.9 K 2023-04-28 04:56:16,981 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/b072903ee5d244cda63c9f20366dc0a2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/b072903ee5d244cda63c9f20366dc0a2 2023-04-28 04:56:16,988 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/b072903ee5d244cda63c9f20366dc0a2, entries=150, sequenceid=360, filesize=11.9 K 2023-04-28 04:56:16,989 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.58 KB/441936, currentSize=26.84 KB/27480 for 17eb93b75efa028513e70ba389775739 in 898ms, sequenceid=360, compaction requested=true 2023-04-28 04:56:16,989 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:16,989 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:16,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:16,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:16,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:16,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:16,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:17,036 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c08a266[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:17,039 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:17,039 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,046 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,047 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:17,047 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:17,047 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:17,047 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,047 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:17,057 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:17,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:17,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:17,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:17,060 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:17,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:17,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:17,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:17,060 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:17,068 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,068 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,068 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:17,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:17,068 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:17,069 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:17,078 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:17,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:17,078 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:17,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:17,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:17,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:17,079 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:17,079 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:17,179 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:17,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:17,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:17,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:17,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:17,183 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:17,183 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:17,183 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:17,183 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:17,184 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,184 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:17,194 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:17,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:17,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,194 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:17,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:17,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:17,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:17,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:17,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:17,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:17,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:17,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:17,196 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:17,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:17,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:17,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:17,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657777269, max timestamp=1682657777296Num uniques 859; 2023-04-28 04:56:17,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:17,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657777269, max timestamp=1682657777296Num uniques 859; 2023-04-28 04:56:17,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:17,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657777269, max timestamp=1682657777296Num uniques 859; 2023-04-28 04:56:17,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:17,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657837328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657777303, max timestamp=1682657777328Num uniques 1118; 2023-04-28 04:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657777303, max timestamp=1682657777328Num uniques 1118; 2023-04-28 04:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-28 04:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657777303, max timestamp=1682657777328Num uniques 1118; 2023-04-28 04:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-28 04:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-28 04:56:17,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657837370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657837370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682657777351, max timestamp=1682657777368Num uniques 1127; 2023-04-28 04:56:17,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-28 04:56:17,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682657777351, max timestamp=1682657777368Num uniques 1127; 2023-04-28 04:56:17,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682657777351, max timestamp=1682657777368Num uniques 1127; 2023-04-28 04:56:17,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-28 04:56:17,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-28 04:56:17,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682657777441, max timestamp=1682657777479Num uniques 1036; 2023-04-28 04:56:17,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-28 04:56:17,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682657777441, max timestamp=1682657777479Num uniques 1036; 2023-04-28 04:56:17,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-28 04:56:17,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657777441, max timestamp=1682657777479Num uniques 1036; 2023-04-28 04:56:17,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-28 04:56:17,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682657777491, max timestamp=1682657777517Num uniques 1195; 2023-04-28 04:56:17,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-28 04:56:17,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682657777491, max timestamp=1682657777517Num uniques 1195; 2023-04-28 04:56:17,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-28 04:56:17,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682657777491, max timestamp=1682657777517Num uniques 1009; 2023-04-28 04:56:17,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15800, suffix heap size=132904,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=345 2023-04-28 04:56:17,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657837565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657837565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657837566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,640 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cbbde4f8a00b4d0180fa3108a2986f14 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cbbde4f8a00b4d0180fa3108a2986f14 2023-04-28 04:56:17,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657837667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657837667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657837667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657837869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657837869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:17,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:17,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657837870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,045 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/bf86a79956b0442580d590674b6e023c 2023-04-28 04:56:18,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:18,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657838172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:18,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657838173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:18,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657838174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,454 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/878faf95a008432cb97d5347e34be873 2023-04-28 04:56:18,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657838680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657838680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:18,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657838680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,865 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/a509815e851c428abde904002257431b 2023-04-28 04:56:18,869 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/bf86a79956b0442580d590674b6e023c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/bf86a79956b0442580d590674b6e023c 2023-04-28 04:56:18,872 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/bf86a79956b0442580d590674b6e023c, entries=100, sequenceid=367, filesize=18.6 K 2023-04-28 04:56:18,873 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/878faf95a008432cb97d5347e34be873 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/878faf95a008432cb97d5347e34be873 2023-04-28 04:56:18,880 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/878faf95a008432cb97d5347e34be873, entries=100, sequenceid=367, filesize=9.5 K 2023-04-28 04:56:18,880 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/a509815e851c428abde904002257431b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/a509815e851c428abde904002257431b 2023-04-28 04:56:18,884 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/a509815e851c428abde904002257431b, entries=100, sequenceid=367, filesize=9.5 K 2023-04-28 04:56:18,885 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 17eb93b75efa028513e70ba389775739 in 1689ms, sequenceid=367, compaction requested=true 2023-04-28 04:56:18,885 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:18,885 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:18,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:18,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:18,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:18,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:18,899 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,899 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:18,899 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,899 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:18,901 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e9b9893[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:18,901 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,909 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:18,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:18,910 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:18,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:18,953 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,953 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:18,953 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:18,954 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:18,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:18,955 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:18,956 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:18,956 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:18,956 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:18,956 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:18,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:18,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:18,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:18,973 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:18,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:18,973 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:18,974 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,983 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:18,983 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:18,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:18,983 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:18,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:18,983 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:18,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:18,983 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:18,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:18,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:18,984 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:18,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,984 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:18,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:18,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:18,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:19,084 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:19,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:19,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:19,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:19,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:19,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:19,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:19,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:19,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:19,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:19,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:19,099 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:19,099 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:19,099 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,099 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:19,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:19,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:19,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:19,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:19,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:19,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:19,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:19,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:19,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:19,101 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=161.02 KB heapSize=447.23 KB 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:19,101 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:19,517 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042806837e4c133e49198018f6a0c9471f4f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042806837e4c133e49198018f6a0c9471f4f 2023-04-28 04:56:19,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:19,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:19,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657839688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657839689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657839692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657839793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657839793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657839795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52052 deadline: 1682657839829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,836 DEBUG [Thread-818] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:19,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:52032 deadline: 1682657839855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:19,858 DEBUG [Thread-809] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:19,954 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=473, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/714ff7d599744095a60507a96eec1df7 2023-04-28 04:56:19,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:19,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657839996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657839996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657839996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657840298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657840301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657840303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,377 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/07efbc44a60f46ddb11ff5e5378e9f21 2023-04-28 04:56:20,391 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2 2023-04-28 04:56:20,397 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/714ff7d599744095a60507a96eec1df7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/714ff7d599744095a60507a96eec1df7 2023-04-28 04:56:20,403 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/714ff7d599744095a60507a96eec1df7, entries=150, sequenceid=473, filesize=25.4 K 2023-04-28 04:56:20,404 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/07efbc44a60f46ddb11ff5e5378e9f21 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07efbc44a60f46ddb11ff5e5378e9f21 2023-04-28 04:56:20,410 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07efbc44a60f46ddb11ff5e5378e9f21, entries=150, sequenceid=473, filesize=11.9 K 2023-04-28 04:56:20,412 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2 2023-04-28 04:56:20,421 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2, entries=150, sequenceid=473, filesize=11.9 K 2023-04-28 04:56:20,422 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~447.19 KB/457920, currentSize=26.84 KB/27480 for 17eb93b75efa028513e70ba389775739 in 1321ms, sequenceid=473, compaction requested=true 2023-04-28 04:56:20,422 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:20,422 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:20,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:20,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:20,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:20,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:20,572 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,572 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:20,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32f064ef[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:20,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:20,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,583 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,583 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:20,583 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:20,584 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,584 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:20,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:20,594 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:20,594 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:20,594 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:20,594 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:20,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:20,601 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:20,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:20,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:20,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:20,602 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:20,614 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,614 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:20,615 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:20,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:20,615 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,617 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,625 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:20,625 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,625 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:20,625 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:20,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:20,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:20,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:20,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:20,626 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:20,626 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:20,625 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:20,626 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,626 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,626 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:20,626 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:20,726 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:20,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:20,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:20,732 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:20,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:20,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:20,732 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:20,733 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,733 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:20,741 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:20,741 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:20,741 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,741 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:20,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:20,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:20,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:20,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:20,744 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:20,744 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:20,744 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:20,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:20,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:20,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:20,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:20,752 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 04:56:20,753 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:20,753 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:20,753 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:20,753 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:20,753 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:20,753 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:20,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657780816, max timestamp=1682657780831Num uniques 859; 2023-04-28 04:56:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:56:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657780816, max timestamp=1682657780831Num uniques 859; 2023-04-28 04:56:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:56:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657780816, max timestamp=1682657780831Num uniques 859; 2023-04-28 04:56:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:56:20,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657840854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657840855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657840856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657840962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657840962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:20,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657840962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657841167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657841168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657841171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,209 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304288e55a7d5ede54e5c894c32b4b8ff983e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304288e55a7d5ede54e5c894c32b4b8ff983e 2023-04-28 04:56:21,245 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=480, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/3cb46d234949450fb1e78b35014464b0 2023-04-28 04:56:21,272 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/7390c2fc91dd47259d94ca5054340d1d 2023-04-28 04:56:21,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657841472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657841472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657841478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,693 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/f06a2141216a4d009fed17ebd300a7c0 2023-04-28 04:56:21,701 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/3cb46d234949450fb1e78b35014464b0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3cb46d234949450fb1e78b35014464b0 2023-04-28 04:56:21,705 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3cb46d234949450fb1e78b35014464b0, entries=150, sequenceid=480, filesize=25.4 K 2023-04-28 04:56:21,706 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/7390c2fc91dd47259d94ca5054340d1d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/7390c2fc91dd47259d94ca5054340d1d 2023-04-28 04:56:21,709 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/7390c2fc91dd47259d94ca5054340d1d, entries=150, sequenceid=480, filesize=11.9 K 2023-04-28 04:56:21,710 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/f06a2141216a4d009fed17ebd300a7c0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/f06a2141216a4d009fed17ebd300a7c0 2023-04-28 04:56:21,714 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/f06a2141216a4d009fed17ebd300a7c0, entries=150, sequenceid=480, filesize=11.9 K 2023-04-28 04:56:21,715 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 17eb93b75efa028513e70ba389775739 in 963ms, sequenceid=480, compaction requested=true 2023-04-28 04:56:21,715 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:21,715 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:21,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:21,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:21,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:21,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:21,761 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,761 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:21,762 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,762 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:21,763 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3284bb2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:21,763 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,783 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,783 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:21,784 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:21,784 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:21,784 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,784 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:21,793 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:21,794 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:21,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:21,796 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:21,796 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:21,797 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:21,797 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:21,797 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:21,804 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,804 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:21,804 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:21,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:21,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:21,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:21,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:21,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:21,815 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:21,815 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:21,815 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:21,815 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:21,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:21,915 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:21,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:21,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:21,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:21,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:21,918 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:21,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:21,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:21,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:21,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:21,962 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:21,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:21,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,962 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:21,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:21,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:21,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:21,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:21,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:21,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:21,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:21,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:21,963 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:21,963 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:21,963 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=161.02 KB heapSize=445.92 KB 2023-04-28 04:56:21,964 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:21,964 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:21,964 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:21,964 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:21,964 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:21,964 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:21,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:21,981 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657841983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657841983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:21,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:21,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657841985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657842089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:22,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657842090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657842292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657842294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,381 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428db9247c146e14202820a782f7bcc1745 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428db9247c146e14202820a782f7bcc1745 2023-04-28 04:56:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657842599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657842601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:22,787 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=517, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/724a4d76af934b9696d9d615649197d9 2023-04-28 04:56:22,813 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/98a3ddf79e6849d2b6049771fc28381d 2023-04-28 04:56:23,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:23,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657843001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:23,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657843102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:23,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657843105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,232 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/4f2db6d2ed2f4cc89d5cb58b42edb624 2023-04-28 04:56:23,237 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/724a4d76af934b9696d9d615649197d9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/724a4d76af934b9696d9d615649197d9 2023-04-28 04:56:23,245 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/724a4d76af934b9696d9d615649197d9, entries=150, sequenceid=517, filesize=25.4 K 2023-04-28 04:56:23,246 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/98a3ddf79e6849d2b6049771fc28381d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/98a3ddf79e6849d2b6049771fc28381d 2023-04-28 04:56:23,251 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/98a3ddf79e6849d2b6049771fc28381d, entries=150, sequenceid=517, filesize=11.9 K 2023-04-28 04:56:23,252 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/4f2db6d2ed2f4cc89d5cb58b42edb624 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4f2db6d2ed2f4cc89d5cb58b42edb624 2023-04-28 04:56:23,256 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4f2db6d2ed2f4cc89d5cb58b42edb624, entries=150, sequenceid=517, filesize=11.9 K 2023-04-28 04:56:23,257 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=26.84 KB/27480 for 17eb93b75efa028513e70ba389775739 in 1293ms, sequenceid=517, compaction requested=true 2023-04-28 04:56:23,257 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:23,257 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:23,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:23,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:23,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:23,257 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:23,309 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,309 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:23,310 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,310 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a1ac223[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:23,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:23,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,330 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,330 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:23,330 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:23,330 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:23,331 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,331 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:23,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:23,341 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:23,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:23,343 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:23,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:23,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:23,343 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:23,343 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:23,351 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,351 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,351 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:23,354 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:23,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,394 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:23,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:23,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:23,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:23,394 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:23,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:23,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:23,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:23,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:23,395 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,395 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:23,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:23,395 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:23,495 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:23,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:23,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:23,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:23,506 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:23,506 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:23,506 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:23,506 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:23,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:23,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:23,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:23,521 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:23,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:23,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:23,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:23,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:23,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:23,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:23,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:23,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:23,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:23,528 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:23,529 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:23,529 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 04:56:23,529 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:23,529 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:23,529 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:23,529 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:23,529 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:23,530 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:23,967 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e04b7cfa414d4ec6bc103427da540edb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e04b7cfa414d4ec6bc103427da540edb 2023-04-28 04:56:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:24,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657784141, max timestamp=1682657784152Num uniques 859; 2023-04-28 04:56:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657784141, max timestamp=1682657784152Num uniques 859; 2023-04-28 04:56:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657784141, max timestamp=1682657784152Num uniques 859; 2023-04-28 04:56:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:24,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657784184, max timestamp=1682657784197Num uniques 1068; 2023-04-28 04:56:24,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 04:56:24,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657784184, max timestamp=1682657784197Num uniques 1068; 2023-04-28 04:56:24,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 04:56:24,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657784184, max timestamp=1682657784197Num uniques 1068; 2023-04-28 04:56:24,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 04:56:24,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657784221, max timestamp=1682657784229Num uniques 1127; 2023-04-28 04:56:24,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-28 04:56:24,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657784221, max timestamp=1682657784229Num uniques 1127; 2023-04-28 04:56:24,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-28 04:56:24,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657784221, max timestamp=1682657784229Num uniques 1127; 2023-04-28 04:56:24,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-28 04:56:24,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682657784265, max timestamp=1682657784279Num uniques 1086; 2023-04-28 04:56:24,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682657784265, max timestamp=1682657784279Num uniques 1086; 2023-04-28 04:56:24,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682657784265, max timestamp=1682657784279Num uniques 1086; 2023-04-28 04:56:24,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-28 04:56:24,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-28 04:56:24,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-28 04:56:24,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:24,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657844321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657784316, max timestamp=1682657784323Num uniques 1145; 2023-04-28 04:56:24,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-28 04:56:24,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657784316, max timestamp=1682657784323Num uniques 1145; 2023-04-28 04:56:24,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-28 04:56:24,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657784316, max timestamp=1682657784323Num uniques 1145; 2023-04-28 04:56:24,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-28 04:56:24,372 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=524, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/3e21a1a18db94bf7b3a207e372fe17ba 2023-04-28 04:56:24,384 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/791c9e2cbdb24e0e90cd0889ec882625 2023-04-28 04:56:24,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657784375, max timestamp=1682657784405Num uniques 1054; 2023-04-28 04:56:24,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 04:56:24,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657784375, max timestamp=1682657784405Num uniques 1054; 2023-04-28 04:56:24,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 04:56:24,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657784375, max timestamp=1682657784405Num uniques 1054; 2023-04-28 04:56:24,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 04:56:24,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682657784432, max timestamp=1682657784450Num uniques 1013; 2023-04-28 04:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682657784432, max timestamp=1682657784450Num uniques 1013; 2023-04-28 04:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-28 04:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-28 04:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1682657784432, max timestamp=1682657784450Num uniques 1013; 2023-04-28 04:56:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=9745, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-28 04:56:24,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1682657784479, max timestamp=1682657784495Num uniques 1072; 2023-04-28 04:56:24,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-04-28 04:56:24,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1682657784479, max timestamp=1682657784495Num uniques 1072; 2023-04-28 04:56:24,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1682657784479, max timestamp=1682657784495Num uniques 1072; 2023-04-28 04:56:24,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-04-28 04:56:24,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-04-28 04:56:24,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682657784541, max timestamp=1682657784558Num uniques 1181; 2023-04-28 04:56:24,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-28 04:56:24,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682657784541, max timestamp=1682657784558Num uniques 1181; 2023-04-28 04:56:24,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-28 04:56:24,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682657784541, max timestamp=1682657784558Num uniques 1181; 2023-04-28 04:56:24,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-28 04:56:24,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682657784575, max timestamp=1682657784607Num uniques 1040; 2023-04-28 04:56:24,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-28 04:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682657784575, max timestamp=1682657784607Num uniques 1040; 2023-04-28 04:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-28 04:56:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682657784575, max timestamp=1682657784607Num uniques 1040; 2023-04-28 04:56:24,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-28 04:56:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682657784640, max timestamp=1682657784655Num uniques 1099; 2023-04-28 04:56:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-28 04:56:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682657784640, max timestamp=1682657784655Num uniques 1099; 2023-04-28 04:56:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-28 04:56:24,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682657784640, max timestamp=1682657784655Num uniques 1099; 2023-04-28 04:56:24,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-28 04:56:24,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:24,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657844696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:24,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657844700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,794 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/752e62b406be40abab43e80b75a06936 2023-04-28 04:56:24,799 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/3e21a1a18db94bf7b3a207e372fe17ba as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3e21a1a18db94bf7b3a207e372fe17ba 2023-04-28 04:56:24,803 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3e21a1a18db94bf7b3a207e372fe17ba, entries=150, sequenceid=524, filesize=25.4 K 2023-04-28 04:56:24,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:24,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657844801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/791c9e2cbdb24e0e90cd0889ec882625 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/791c9e2cbdb24e0e90cd0889ec882625 2023-04-28 04:56:24,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:24,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657844805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,813 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/791c9e2cbdb24e0e90cd0889ec882625, entries=150, sequenceid=524, filesize=11.9 K 2023-04-28 04:56:24,813 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/752e62b406be40abab43e80b75a06936 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/752e62b406be40abab43e80b75a06936 2023-04-28 04:56:24,819 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/752e62b406be40abab43e80b75a06936, entries=150, sequenceid=524, filesize=11.9 K 2023-04-28 04:56:24,826 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for 17eb93b75efa028513e70ba389775739 in 1297ms, sequenceid=524, compaction requested=true 2023-04-28 04:56:24,826 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:24,826 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:24,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:24,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:24,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:24,826 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:24,878 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,878 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:24,878 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,878 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b798f3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:24,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:24,880 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,888 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,888 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:24,888 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:24,888 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:24,889 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,889 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:24,899 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:24,899 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:24,899 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,899 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:24,901 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:24,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:24,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:24,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:24,902 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:24,909 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,909 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:24,909 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:24,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:24,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:24,911 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:24,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:24,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:24,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:24,920 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:24,920 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:24,920 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:24,921 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:24,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:25,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=161.02 KB heapSize=452.25 KB 2023-04-28 04:56:25,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:25,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:25,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:25,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:25,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:25,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:25,021 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:25,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657845015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:25,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:25,023 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:25,023 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,023 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:25,023 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:25,023 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657845019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657845020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,026 DEBUG [Thread-816] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:25,035 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,035 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,036 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,036 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:25,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:25,037 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,037 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:25,037 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:25,037 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:25,037 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:25,046 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,046 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,046 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,046 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:25,047 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d14816[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:25,047 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,056 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:25,057 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,057 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:25,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:25,057 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:25,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657845122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657845125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:25,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,149 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:25,151 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:25,151 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:25,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:25,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:25,152 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:25,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,225 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:25,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:25,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,225 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,225 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,225 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,325 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:25,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:25,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:25,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657845326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:25,328 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:25,328 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,328 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:25,328 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:25,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657845329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,341 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,341 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,341 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,341 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:25,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:25,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:25,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:25,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:25,342 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:25,342 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,343 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:25,343 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:25,343 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:25,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:25,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:25,343 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:25,351 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,351 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,351 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,352 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:25,353 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e5eb45[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:25,353 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,362 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,362 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:25,362 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:25,362 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:25,362 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,362 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:25,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:25,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,372 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:25,375 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:25,375 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:25,375 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:25,375 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:25,375 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:25,383 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,383 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,383 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:25,383 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,384 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,421 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428506df3da70454765801b3de0db24808b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428506df3da70454765801b3de0db24808b 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:25,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:25,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,468 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:25,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:25,468 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,568 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:25,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:25,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:25,571 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:25,571 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,571 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:25,571 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:25,572 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,572 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,612 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,613 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,613 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:25,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:25,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:25,614 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,614 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:25,614 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:25,614 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:25,614 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:25,623 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,623 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,623 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,623 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:25,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54123fa0[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:25,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657845630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,633 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,633 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,633 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,634 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:25,634 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:25,634 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:25,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:25,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657845631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,678 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:25,678 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,678 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,678 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:25,680 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:25,681 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:25,681 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:25,681 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:25,681 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:25,686 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,686 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:25,686 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,687 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,696 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:25,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:25,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:25,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,697 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,697 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:25,697 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,697 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:25,797 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:25,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:25,800 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:25,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:25,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:25,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,812 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:25,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:25,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:25,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:25,814 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,814 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:25,814 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:25,814 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:25,814 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:25,823 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,823 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,823 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,823 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:25,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300655d1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:25,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,827 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=734, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/dfd8a58ec7d641379972ba098e22c2b0 2023-04-28 04:56:25,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:25,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:25,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:25,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:25,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:25,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,979 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:25,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:25,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:25,982 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:25,982 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:25,982 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:25,982 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:25,982 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:26,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,002 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,003 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:26,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:26,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:26,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:26,012 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:26,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:26,013 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:26,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,013 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:26,013 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,013 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,013 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,013 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,013 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:26,113 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:26,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:26,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:26,128 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:26,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:26,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:26,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:26,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657846134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657846138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,181 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,181 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:26,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:26,181 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,181 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:26,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:26,182 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:26,182 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:26,182 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:26,182 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:26,182 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:26,182 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,183 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:26,183 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:26,183 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:26,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:26,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:26,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:26,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,191 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,192 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,192 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53742a3f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:26,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:26,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,202 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,202 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:26,202 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:26,202 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:26,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:26,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:26,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:26,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:26,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:26,215 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:26,215 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:26,215 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:26,215 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:26,215 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:26,223 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,223 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,223 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:26,223 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,225 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:26,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:26,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:26,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:26,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:26,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:26,234 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:26,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:26,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,234 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:26,234 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,234 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,234 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,234 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,234 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:26,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/07b973e91361425db13480edc773b15d 2023-04-28 04:56:26,334 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:26,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:26,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:26,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:26,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:26,337 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:26,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:26,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:26,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:26,338 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,338 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,466 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:26,466 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:26,467 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,467 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:26,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:26,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:26,467 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:26,467 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:26,467 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:26,468 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,468 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:26,468 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:26,468 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:26,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:26,486 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,486 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,486 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,486 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54cc5cd7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:26,488 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:26,488 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,496 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:26,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:26,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:26,497 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,497 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:26,507 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,507 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:26,507 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,507 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:26,509 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:26,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:26,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:26,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:26,509 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:26,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:26,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,518 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:26,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:26,518 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,519 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:26,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:26,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:26,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:26,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:26,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:26,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,529 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,529 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:26,529 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,529 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:26,529 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:26,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:26,629 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:26,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:26,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:26,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:26,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:26,631 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:26,631 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:26,631 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:26,631 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:26,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/517d2f53fbff45a79003cae79600c9a1 2023-04-28 04:56:26,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/dfd8a58ec7d641379972ba098e22c2b0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/dfd8a58ec7d641379972ba098e22c2b0 2023-04-28 04:56:26,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/dfd8a58ec7d641379972ba098e22c2b0, entries=200, sequenceid=734, filesize=32.3 K 2023-04-28 04:56:26,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/07b973e91361425db13480edc773b15d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07b973e91361425db13480edc773b15d 2023-04-28 04:56:26,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07b973e91361425db13480edc773b15d, entries=150, sequenceid=734, filesize=11.9 K 2023-04-28 04:56:26,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/517d2f53fbff45a79003cae79600c9a1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/517d2f53fbff45a79003cae79600c9a1 2023-04-28 04:56:26,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/517d2f53fbff45a79003cae79600c9a1, entries=150, sequenceid=734, filesize=11.9 K 2023-04-28 04:56:26,670 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~452.20 KB/463056, currentSize=20.13 KB/20610 for 17eb93b75efa028513e70ba389775739 in 1662ms, sequenceid=734, compaction requested=true 2023-04-28 04:56:26,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:26,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:26,671 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-28 04:56:26,671 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-28 04:56:26,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:26,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:26,673 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107020 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-28 04:56:26,673 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/B is initiating minor compaction (all files) 2023-04-28 04:56:26,674 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 241186 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-28 04:56:26,674 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/B in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,674 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/A is initiating minor compaction (all files) 2023-04-28 04:56:26,674 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/A in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,674 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/867cf39946654123aa202d227b193f0f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/359da8f89c2e47e4bb68069c1317ec75, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d283ef9931a14438baf746b02cd084bf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/878faf95a008432cb97d5347e34be873, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07efbc44a60f46ddb11ff5e5378e9f21, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/7390c2fc91dd47259d94ca5054340d1d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/98a3ddf79e6849d2b6049771fc28381d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/791c9e2cbdb24e0e90cd0889ec882625, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07b973e91361425db13480edc773b15d] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=104.5 K 2023-04-28 04:56:26,674 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c654e8ecc20e4659a1c98647e463e87a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/a0bf57e4abfb4e9e885b614fb70c761d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5db41099cbb442b78592e07c610af955, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/bf86a79956b0442580d590674b6e023c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/714ff7d599744095a60507a96eec1df7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3cb46d234949450fb1e78b35014464b0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/724a4d76af934b9696d9d615649197d9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3e21a1a18db94bf7b3a207e372fe17ba, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/dfd8a58ec7d641379972ba098e22c2b0] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=235.5 K 2023-04-28 04:56:26,674 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c654e8ecc20e4659a1c98647e463e87a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682657773768 2023-04-28 04:56:26,674 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 867cf39946654123aa202d227b193f0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682657773768 2023-04-28 04:56:26,675 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting a0bf57e4abfb4e9e885b614fb70c761d, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657774402 2023-04-28 04:56:26,675 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 359da8f89c2e47e4bb68069c1317ec75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657774420 2023-04-28 04:56:26,675 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5db41099cbb442b78592e07c610af955, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682657775510 2023-04-28 04:56:26,675 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d283ef9931a14438baf746b02cd084bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682657775510 2023-04-28 04:56:26,675 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting bf86a79956b0442580d590674b6e023c, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682657776132 2023-04-28 04:56:26,675 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 878faf95a008432cb97d5347e34be873, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682657776132 2023-04-28 04:56:26,676 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 714ff7d599744095a60507a96eec1df7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682657777564 2023-04-28 04:56:26,676 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 07efbc44a60f46ddb11ff5e5378e9f21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682657777564 2023-04-28 04:56:26,676 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3cb46d234949450fb1e78b35014464b0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682657779686 2023-04-28 04:56:26,676 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7390c2fc91dd47259d94ca5054340d1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682657779686 2023-04-28 04:56:26,676 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 724a4d76af934b9696d9d615649197d9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682657780850 2023-04-28 04:56:26,676 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 98a3ddf79e6849d2b6049771fc28381d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682657780850 2023-04-28 04:56:26,677 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3e21a1a18db94bf7b3a207e372fe17ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1682657781981 2023-04-28 04:56:26,677 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 791c9e2cbdb24e0e90cd0889ec882625, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1682657781981 2023-04-28 04:56:26,677 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting dfd8a58ec7d641379972ba098e22c2b0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682657784689 2023-04-28 04:56:26,677 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 07b973e91361425db13480edc773b15d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682657784695 2023-04-28 04:56:26,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:26,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:26,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:26,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:26,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:26,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:26,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:26,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:26,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:26,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:26,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:26,681 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:26,681 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:26,681 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:26,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:26,684 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:26,684 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:26,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:26,684 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 04:56:26,685 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:26,685 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:26,685 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:26,685 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:26,685 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:26,685 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:26,711 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#B#compaction#151 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:26,717 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#A#compaction#150 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:27,122 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428336bce030c574de9827a6ef1153592e7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428336bce030c574de9827a6ef1153592e7 2023-04-28 04:56:27,125 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/4c6fbe0c76a0476fb0c1afa1e1e2a163 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/4c6fbe0c76a0476fb0c1afa1e1e2a163 2023-04-28 04:56:27,133 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 17eb93b75efa028513e70ba389775739/B of 17eb93b75efa028513e70ba389775739 into 4c6fbe0c76a0476fb0c1afa1e1e2a163(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:27,133 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:27,133 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/B, priority=7, startTime=1682657786671; duration=0sec 2023-04-28 04:56:27,133 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:27,133 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-28 04:56:27,137 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107020 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-28 04:56:27,137 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/C is initiating minor compaction (all files) 2023-04-28 04:56:27,137 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/C in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:27,137 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/516be6a7aa9a402883e3803dd3301f1a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/573cc5737ce14e808b1283bfe43e051f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/b072903ee5d244cda63c9f20366dc0a2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/a509815e851c428abde904002257431b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/f06a2141216a4d009fed17ebd300a7c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4f2db6d2ed2f4cc89d5cb58b42edb624, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/752e62b406be40abab43e80b75a06936, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/517d2f53fbff45a79003cae79600c9a1] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=104.5 K 2023-04-28 04:56:27,138 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 516be6a7aa9a402883e3803dd3301f1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682657773768 2023-04-28 04:56:27,139 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 573cc5737ce14e808b1283bfe43e051f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657774420 2023-04-28 04:56:27,139 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b072903ee5d244cda63c9f20366dc0a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1682657775510 2023-04-28 04:56:27,140 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a509815e851c428abde904002257431b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682657776132 2023-04-28 04:56:27,140 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 056bbd3cbcaf4ddeb015fd3e10bcb5e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682657777564 2023-04-28 04:56:27,140 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting f06a2141216a4d009fed17ebd300a7c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682657779686 2023-04-28 04:56:27,141 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4f2db6d2ed2f4cc89d5cb58b42edb624, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1682657780850 2023-04-28 04:56:27,141 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 752e62b406be40abab43e80b75a06936, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1682657781981 2023-04-28 04:56:27,141 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 517d2f53fbff45a79003cae79600c9a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682657784695 2023-04-28 04:56:27,160 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=740, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/7b29eadd084a43cd8206f65fefa28bfa 2023-04-28 04:56:27,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:27,175 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#C#compaction#152 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:27,180 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/d01dcf96d200449e811e16955ba674ed 2023-04-28 04:56:27,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657787213, max timestamp=1682657787227Num uniques 859; 2023-04-28 04:56:27,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657787213, max timestamp=1682657787227Num uniques 859; 2023-04-28 04:56:27,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657787213, max timestamp=1682657787227Num uniques 859; 2023-04-28 04:56:27,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:27,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:27,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682657787250, max timestamp=1682657787278Num uniques 1068; 2023-04-28 04:56:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682657787250, max timestamp=1682657787278Num uniques 1068; 2023-04-28 04:56:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682657787250, max timestamp=1682657787278Num uniques 1068; 2023-04-28 04:56:27,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-28 04:56:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-28 04:56:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-28 04:56:27,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657847329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657787315, max timestamp=1682657787335Num uniques 1227; 2023-04-28 04:56:27,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-28 04:56:27,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657787315, max timestamp=1682657787335Num uniques 1227; 2023-04-28 04:56:27,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-28 04:56:27,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657787315, max timestamp=1682657787335Num uniques 1227; 2023-04-28 04:56:27,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-28 04:56:27,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657787385, max timestamp=1682657787411Num uniques 1086; 2023-04-28 04:56:27,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 04:56:27,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657787385, max timestamp=1682657787411Num uniques 1086; 2023-04-28 04:56:27,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 04:56:27,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657787385, max timestamp=1682657787411Num uniques 1086; 2023-04-28 04:56:27,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 04:56:27,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682657787454, max timestamp=1682657787466Num uniques 1095; 2023-04-28 04:56:27,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 04:56:27,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682657787454, max timestamp=1682657787466Num uniques 1095; 2023-04-28 04:56:27,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 04:56:27,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657787454, max timestamp=1682657787466Num uniques 1095; 2023-04-28 04:56:27,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=6870, suffix heap size=141848,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=150 2023-04-28 04:56:27,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657787495, max timestamp=1682657787501Num uniques 1104; 2023-04-28 04:56:27,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 04:56:27,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657787495, max timestamp=1682657787501Num uniques 1104; 2023-04-28 04:56:27,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 04:56:27,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657787495, max timestamp=1682657787501Num uniques 1009; 2023-04-28 04:56:27,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=9340, suffix heap size=132904,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=204 2023-04-28 04:56:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1682657787513, max timestamp=1682657787539Num uniques 1063; 2023-04-28 04:56:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=16618, suffix heap size=138520,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=363 2023-04-28 04:56:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1682657787513, max timestamp=1682657787539Num uniques 1063; 2023-04-28 04:56:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=16618, suffix heap size=138520,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=363 2023-04-28 04:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1682657787513, max timestamp=1682657787539Num uniques 1063; 2023-04-28 04:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=16618, suffix heap size=138520,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=363 2023-04-28 04:56:27,571 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/ec6e75da43bf46b3ae70967729417b93 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ec6e75da43bf46b3ae70967729417b93 2023-04-28 04:56:27,576 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 17eb93b75efa028513e70ba389775739/A of 17eb93b75efa028513e70ba389775739 into ec6e75da43bf46b3ae70967729417b93(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:27,576 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:27,576 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/A, priority=7, startTime=1682657786671; duration=0sec 2023-04-28 04:56:27,577 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682657787571, max timestamp=1682657787585Num uniques 1222; 2023-04-28 04:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682657787571, max timestamp=1682657787585Num uniques 1222; 2023-04-28 04:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=10162, suffix heap size=155056,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=222 2023-04-28 04:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=10162, suffix heap size=155056,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=222 2023-04-28 04:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682657787571, max timestamp=1682657787585Num uniques 1222; 2023-04-28 04:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=10162, suffix heap size=155056,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=222 2023-04-28 04:56:27,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1682657787600, max timestamp=1682657787618Num uniques 1081; 2023-04-28 04:56:27,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-28 04:56:27,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1682657787600, max timestamp=1682657787618Num uniques 1081; 2023-04-28 04:56:27,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-28 04:56:27,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1682657787600, max timestamp=1682657787618Num uniques 1081; 2023-04-28 04:56:27,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-28 04:56:27,628 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/e14b742bf4b14f0baf3cc5486ac27b7a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/e14b742bf4b14f0baf3cc5486ac27b7a 2023-04-28 04:56:27,635 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/2564f094634f4653b0e3bf0722540dbc 2023-04-28 04:56:27,638 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 17eb93b75efa028513e70ba389775739/C of 17eb93b75efa028513e70ba389775739 into e14b742bf4b14f0baf3cc5486ac27b7a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:27,638 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:27,638 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/C, priority=7, startTime=1682657786671; duration=0sec 2023-04-28 04:56:27,638 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:27,639 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/7b29eadd084a43cd8206f65fefa28bfa as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/7b29eadd084a43cd8206f65fefa28bfa 2023-04-28 04:56:27,643 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/7b29eadd084a43cd8206f65fefa28bfa, entries=100, sequenceid=740, filesize=18.6 K 2023-04-28 04:56:27,644 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/d01dcf96d200449e811e16955ba674ed as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d01dcf96d200449e811e16955ba674ed 2023-04-28 04:56:27,648 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d01dcf96d200449e811e16955ba674ed, entries=100, sequenceid=740, filesize=9.5 K 2023-04-28 04:56:27,649 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/2564f094634f4653b0e3bf0722540dbc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/2564f094634f4653b0e3bf0722540dbc 2023-04-28 04:56:27,656 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/2564f094634f4653b0e3bf0722540dbc, entries=100, sequenceid=740, filesize=9.5 K 2023-04-28 04:56:27,657 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=120.76 KB/123660 for 17eb93b75efa028513e70ba389775739 in 973ms, sequenceid=740, compaction requested=false 2023-04-28 04:56:27,657 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:27,657 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:27,657 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:27,657 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:27,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:27,658 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:27,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=127.47 KB heapSize=348.94 KB 2023-04-28 04:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:27,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:27,665 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:27,665 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:27,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116686d1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:27,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:27,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,675 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:27,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:27,676 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,676 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:27,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:27,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657847677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657847678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,686 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,686 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:27,686 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:27,686 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:27,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:27,689 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:27,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:27,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:27,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:27,689 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:27,696 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,696 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:27,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,696 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:27,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:27,697 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,698 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:27,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:27,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:27,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:27,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:27,707 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:27,707 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:27,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,708 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,708 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,708 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:27,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657847781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657847782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,807 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:27,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:27,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:27,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:27,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:27,811 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:27,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:27,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:27,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:27,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,823 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:27,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:27,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:27,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:27,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:27,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:27,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:27,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:27,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:27,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:27,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:27,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:27,824 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:27,824 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:27,825 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:27,825 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,825 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:27,825 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:27,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,834 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,834 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@134cf1f1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:27,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:27,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,844 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:27,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:27,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:27,844 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,844 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:27,854 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:27,854 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:27,854 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:27,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:27,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:27,861 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:27,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:27,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:27,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:27,861 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:27,865 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,865 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:27,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,865 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:27,866 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,875 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:27,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:27,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:27,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:27,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:27,876 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:27,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:27,876 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:27,876 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,876 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:27,876 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:27,876 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:27,976 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:27,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:27,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:27,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:27,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:27,979 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:27,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657847983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657847984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,991 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:27,991 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:27,991 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:27,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:27,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:27,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:27,993 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:27,993 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:27,993 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:27,993 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:27,993 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:28,028 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,028 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:28,029 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,029 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619b1475[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:28,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:28,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,044 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:28,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:28,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:28,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:28,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:28,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:28,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:28,057 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:28,058 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:28,058 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:28,058 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:28,058 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:28,065 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,065 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,065 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:28,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:28,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:28,066 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,067 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,075 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:28,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:28,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,076 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:28,076 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,076 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:28,077 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:28,080 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428be0e10d1114545a4bbc5190e8b8a9001 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428be0e10d1114545a4bbc5190e8b8a9001 2023-04-28 04:56:28,096 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=914, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/01af7fd56ff34141aace722aad7e5cb8 2023-04-28 04:56:28,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/6c531b33488e4a2ca09a7edeb1a07993 2023-04-28 04:56:28,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/04c81ab1b8994622ad3dd99e2f7d7703 2023-04-28 04:56:28,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/01af7fd56ff34141aace722aad7e5cb8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/01af7fd56ff34141aace722aad7e5cb8 2023-04-28 04:56:28,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/01af7fd56ff34141aace722aad7e5cb8, entries=150, sequenceid=914, filesize=25.4 K 2023-04-28 04:56:28,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/6c531b33488e4a2ca09a7edeb1a07993 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/6c531b33488e4a2ca09a7edeb1a07993 2023-04-28 04:56:28,151 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/6c531b33488e4a2ca09a7edeb1a07993, entries=150, sequenceid=914, filesize=11.9 K 2023-04-28 04:56:28,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/04c81ab1b8994622ad3dd99e2f7d7703 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/04c81ab1b8994622ad3dd99e2f7d7703 2023-04-28 04:56:28,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/04c81ab1b8994622ad3dd99e2f7d7703, entries=150, sequenceid=914, filesize=11.9 K 2023-04-28 04:56:28,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~368.81 KB/377664, currentSize=53.67 KB/54960 for 17eb93b75efa028513e70ba389775739 in 500ms, sequenceid=914, compaction requested=true 2023-04-28 04:56:28,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:28,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:28,159 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:28,159 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:28,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:28,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:28,160 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71636 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:28,160 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/A is initiating minor compaction (all files) 2023-04-28 04:56:28,160 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:28,160 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/A in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:28,160 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/B is initiating minor compaction (all files) 2023-04-28 04:56:28,160 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/B in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:28,160 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/4c6fbe0c76a0476fb0c1afa1e1e2a163, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d01dcf96d200449e811e16955ba674ed, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/6c531b33488e4a2ca09a7edeb1a07993] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=33.8 K 2023-04-28 04:56:28,160 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ec6e75da43bf46b3ae70967729417b93, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/7b29eadd084a43cd8206f65fefa28bfa, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/01af7fd56ff34141aace722aad7e5cb8] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=70.0 K 2023-04-28 04:56:28,161 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ec6e75da43bf46b3ae70967729417b93, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682657784695 2023-04-28 04:56:28,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4c6fbe0c76a0476fb0c1afa1e1e2a163, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682657784695 2023-04-28 04:56:28,161 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7b29eadd084a43cd8206f65fefa28bfa, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682657785012 2023-04-28 04:56:28,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d01dcf96d200449e811e16955ba674ed, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682657785012 2023-04-28 04:56:28,161 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 01af7fd56ff34141aace722aad7e5cb8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682657787632 2023-04-28 04:56:28,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 6c531b33488e4a2ca09a7edeb1a07993, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682657787632 2023-04-28 04:56:28,175 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#B#compaction#159 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:28,176 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:28,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:28,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:28,178 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#A#compaction#158 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:28,179 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:28,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:28,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:28,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:28,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:28,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,192 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:28,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:28,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:28,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:28,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:28,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:28,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:28,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:28,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:28,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:28,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. started... 2023-04-28 04:56:28,195 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:28,195 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:28,196 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:28,204 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/460388449ac84963b8d1b78befae864f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/460388449ac84963b8d1b78befae864f 2023-04-28 04:56:28,210 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/B of 17eb93b75efa028513e70ba389775739 into 460388449ac84963b8d1b78befae864f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:28,210 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:28,210 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/B, priority=13, startTime=1682657788159; duration=0sec 2023-04-28 04:56:28,210 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:28,210 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:28,211 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:28,212 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/C is initiating minor compaction (all files) 2023-04-28 04:56:28,212 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/C in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:28,212 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/e14b742bf4b14f0baf3cc5486ac27b7a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/2564f094634f4653b0e3bf0722540dbc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/04c81ab1b8994622ad3dd99e2f7d7703] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=33.8 K 2023-04-28 04:56:28,212 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting e14b742bf4b14f0baf3cc5486ac27b7a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682657784695 2023-04-28 04:56:28,213 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 2564f094634f4653b0e3bf0722540dbc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682657785012 2023-04-28 04:56:28,214 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 04c81ab1b8994622ad3dd99e2f7d7703, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682657787632 2023-04-28 04:56:28,231 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:28,238 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c8477a5ccc3f48bfbd41ebb51020f662 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c8477a5ccc3f48bfbd41ebb51020f662 2023-04-28 04:56:28,255 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/1d6c2ca22581450d8193c36930dfa913 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/1d6c2ca22581450d8193c36930dfa913 2023-04-28 04:56:28,260 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/C of 17eb93b75efa028513e70ba389775739 into 1d6c2ca22581450d8193c36930dfa913(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:28,260 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:28,260 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/C, priority=13, startTime=1682657788159; duration=0sec 2023-04-28 04:56:28,260 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:28,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:28,290 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. as already flushing 2023-04-28 04:56:28,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657788314, max timestamp=1682657788326Num uniques 859; 2023-04-28 04:56:28,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:28,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657788314, max timestamp=1682657788326Num uniques 859; 2023-04-28 04:56:28,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657788314, max timestamp=1682657788326Num uniques 859; 2023-04-28 04:56:28,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:28,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:28,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657848354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657848354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657848458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657848458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,629 DEBUG [Thread-823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x651f745a to 127.0.0.1:52932 2023-04-28 04:56:28,629 DEBUG [Thread-823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:28,630 DEBUG [Thread-827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3432fa3f to 127.0.0.1:52932 2023-04-28 04:56:28,630 DEBUG [Thread-827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:28,631 DEBUG [Thread-825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d429358 to 127.0.0.1:52932 2023-04-28 04:56:28,631 DEBUG [Thread-825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:28,632 DEBUG [Thread-821] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d5bdceb to 127.0.0.1:52932 2023-04-28 04:56:28,632 DEBUG [Thread-821] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:28,652 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=925, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/6918d024b3444bb297dbd97a450dd5a1 2023-04-28 04:56:28,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657848661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657848661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657848964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657848964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:29,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52060 deadline: 1682657849060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,061 DEBUG [Thread-816] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8205 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:29,062 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/973b74cf14314e06ac68dc32331222d3 2023-04-28 04:56:29,412 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/c913d814b8e847f4928bd662b44d28f0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c913d814b8e847f4928bd662b44d28f0 2023-04-28 04:56:29,418 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/A of 17eb93b75efa028513e70ba389775739 into c913d814b8e847f4928bd662b44d28f0(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:56:29,418 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:29,418 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/A, priority=13, startTime=1682657788159; duration=1sec 2023-04-28 04:56:29,419 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:29,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:29,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52012 deadline: 1682657849465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:29,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:52016 deadline: 1682657849469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,473 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/ca2c4d2cc5274f2181489e4857da052e 2023-04-28 04:56:29,481 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/6918d024b3444bb297dbd97a450dd5a1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/6918d024b3444bb297dbd97a450dd5a1 2023-04-28 04:56:29,488 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/6918d024b3444bb297dbd97a450dd5a1, entries=150, sequenceid=925, filesize=25.4 K 2023-04-28 04:56:29,490 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/973b74cf14314e06ac68dc32331222d3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/973b74cf14314e06ac68dc32331222d3 2023-04-28 04:56:29,495 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/973b74cf14314e06ac68dc32331222d3, entries=150, sequenceid=925, filesize=11.9 K 2023-04-28 04:56:29,495 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/ca2c4d2cc5274f2181489e4857da052e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/ca2c4d2cc5274f2181489e4857da052e 2023-04-28 04:56:29,499 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/ca2c4d2cc5274f2181489e4857da052e, entries=150, sequenceid=925, filesize=11.9 K 2023-04-28 04:56:29,500 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 17eb93b75efa028513e70ba389775739 in 1305ms, sequenceid=925, compaction requested=false 2023-04-28 04:56:29,500 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:29,500 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:29,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:29,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:29,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:29,500 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:29,507 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,507 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:29,507 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,507 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5787d675[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:29,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:29,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,518 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:29,518 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:29,518 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:29,518 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,518 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:29,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:29,528 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:29,528 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:29,532 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:29,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:29,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:29,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:29,532 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:29,538 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,538 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,539 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:29,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:29,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:29,539 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:29,542 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:29,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:29,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:29,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:29,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:29,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:29,550 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:29,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:29,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:29,549 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:29,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:29,550 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,550 DEBUG [Thread-820] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:29,551 DEBUG [Thread-820] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:29,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:29,550 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:29,551 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:29,651 DEBUG [Thread-820] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:29,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:29,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 04:56:29,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:29,882 DEBUG [Thread-818] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50feca84 to 127.0.0.1:52932 2023-04-28 04:56:29,882 DEBUG [Thread-818] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:29,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:29,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:29,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:29,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:29,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:29,943 DEBUG [Thread-809] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d781def to 127.0.0.1:52932 2023-04-28 04:56:29,943 DEBUG [Thread-809] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:30,306 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042844494754bbe84df7b2a09bcd79f2e7fe to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042844494754bbe84df7b2a09bcd79f2e7fe 2023-04-28 04:56:30,472 DEBUG [Thread-811] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7acc1387 to 127.0.0.1:52932 2023-04-28 04:56:30,472 DEBUG [Thread-811] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:30,476 DEBUG [Thread-813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7be739cc to 127.0.0.1:52932 2023-04-28 04:56:30,476 DEBUG [Thread-813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:30,717 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=963, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/011c94d5d28142758e732d97ab23d5c5 2023-04-28 04:56:31,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=963 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/5ec1724379584db283bc2a3e515af04b 2023-04-28 04:56:31,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=963 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/89140c4d5958494b8db2acc7ba4d2ed9 2023-04-28 04:56:31,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/011c94d5d28142758e732d97ab23d5c5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/011c94d5d28142758e732d97ab23d5c5 2023-04-28 04:56:31,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/011c94d5d28142758e732d97ab23d5c5, entries=150, sequenceid=963, filesize=25.4 K 2023-04-28 04:56:31,561 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/5ec1724379584db283bc2a3e515af04b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/5ec1724379584db283bc2a3e515af04b 2023-04-28 04:56:31,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/5ec1724379584db283bc2a3e515af04b, entries=150, sequenceid=963, filesize=11.9 K 2023-04-28 04:56:31,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/89140c4d5958494b8db2acc7ba4d2ed9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/89140c4d5958494b8db2acc7ba4d2ed9 2023-04-28 04:56:31,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/89140c4d5958494b8db2acc7ba4d2ed9, entries=150, sequenceid=963, filesize=11.9 K 2023-04-28 04:56:31,571 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=20.13 KB/20610 for 17eb93b75efa028513e70ba389775739 in 1690ms, sequenceid=963, compaction requested=true 2023-04-28 04:56:31,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:31,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:31,572 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:31,572 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:31,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:31,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:31,573 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:31,574 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/A is initiating minor compaction (all files) 2023-04-28 04:56:31,574 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/A in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:31,574 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:31,574 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c913d814b8e847f4928bd662b44d28f0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/6918d024b3444bb297dbd97a450dd5a1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/011c94d5d28142758e732d97ab23d5c5] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=76.9 K 2023-04-28 04:56:31,574 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/B is initiating minor compaction (all files) 2023-04-28 04:56:31,574 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/B in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:31,574 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/460388449ac84963b8d1b78befae864f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/973b74cf14314e06ac68dc32331222d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/5ec1724379584db283bc2a3e515af04b] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=36.3 K 2023-04-28 04:56:31,574 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c913d814b8e847f4928bd662b44d28f0, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682657787632 2023-04-28 04:56:31,575 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 460388449ac84963b8d1b78befae864f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682657787632 2023-04-28 04:56:31,575 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6918d024b3444bb297dbd97a450dd5a1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=925, earliestPutTs=1682657787670 2023-04-28 04:56:31,575 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 973b74cf14314e06ac68dc32331222d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=925, earliestPutTs=1682657787670 2023-04-28 04:56:31,576 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 011c94d5d28142758e732d97ab23d5c5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1682657788336 2023-04-28 04:56:31,576 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5ec1724379584db283bc2a3e515af04b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1682657788336 2023-04-28 04:56:31,593 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#B#compaction#167 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:31,606 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#A#compaction#168 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:31,610 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/0013147da5d443dcaefd4d161e040d8b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/0013147da5d443dcaefd4d161e040d8b 2023-04-28 04:56:31,614 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/B of 17eb93b75efa028513e70ba389775739 into 0013147da5d443dcaefd4d161e040d8b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:31,614 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:31,614 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/B, priority=13, startTime=1682657791572; duration=0sec 2023-04-28 04:56:31,614 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:31,614 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:31,615 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:31,615 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 17eb93b75efa028513e70ba389775739/C is initiating minor compaction (all files) 2023-04-28 04:56:31,615 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 17eb93b75efa028513e70ba389775739/C in TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:31,615 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/1d6c2ca22581450d8193c36930dfa913, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/ca2c4d2cc5274f2181489e4857da052e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/89140c4d5958494b8db2acc7ba4d2ed9] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp, totalSize=36.3 K 2023-04-28 04:56:31,616 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1d6c2ca22581450d8193c36930dfa913, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682657787632 2023-04-28 04:56:31,616 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ca2c4d2cc5274f2181489e4857da052e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=925, earliestPutTs=1682657787670 2023-04-28 04:56:31,616 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 89140c4d5958494b8db2acc7ba4d2ed9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=963, earliestPutTs=1682657788336 2023-04-28 04:56:31,624 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 17eb93b75efa028513e70ba389775739#C#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:32,038 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/340bb7213e434dad925f4b5b16ce98d3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/340bb7213e434dad925f4b5b16ce98d3 2023-04-28 04:56:32,043 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/C of 17eb93b75efa028513e70ba389775739 into 340bb7213e434dad925f4b5b16ce98d3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:32,043 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:32,044 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/C, priority=13, startTime=1682657791572; duration=0sec 2023-04-28 04:56:32,044 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:32,828 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/ba26277961f44a21bdc4ac4b15574eab as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ba26277961f44a21bdc4ac4b15574eab 2023-04-28 04:56:32,834 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 17eb93b75efa028513e70ba389775739/A of 17eb93b75efa028513e70ba389775739 into ba26277961f44a21bdc4ac4b15574eab(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:56:32,834 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:32,834 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739., storeName=17eb93b75efa028513e70ba389775739/A, priority=13, startTime=1682657791572; duration=1sec 2023-04-28 04:56:32,834 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:39,093 DEBUG [Thread-816] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x486fba63 to 127.0.0.1:52932 2023-04-28 04:56:39,093 DEBUG [Thread-816] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 321 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 361 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-28 04:56:39,094 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 4165 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 4144 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1810 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5430 rows 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1824 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5472 rows 2023-04-28 04:56:39,095 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:56:39,095 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x671fd08e to 127.0.0.1:52932 2023-04-28 04:56:39,095 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:56:39,097 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 04:56:39,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$11(2418): Client=jenkins//136.243.104.168 disable TestAcidGuarantees 2023-04-28 04:56:39,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:39,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 04:56:39,103 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657799103"}]},"ts":"1682657799103"} 2023-04-28 04:56:39,104 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 04:56:39,148 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 04:56:39,149 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, UNASSIGN}] 2023-04-28 04:56:39,151 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, UNASSIGN 2023-04-28 04:56:39,151 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:39,151 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657799151"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657799151"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657799151"}]},"ts":"1682657799151"} 2023-04-28 04:56:39,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:56:39,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 04:56:39,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:39,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 17eb93b75efa028513e70ba389775739, disabling compactions & flushes 2023-04-28 04:56:39,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:39,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. after waiting 0 ms 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:39,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing 17eb93b75efa028513e70ba389775739 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=A 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=B 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 17eb93b75efa028513e70ba389775739, store=C 2023-04-28 04:56:39,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:39,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 04:56:39,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 04:56:39,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042856eb16ada0154f91886c50853151e208 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042856eb16ada0154f91886c50853151e208 2023-04-28 04:56:40,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=973, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/590a4a927c944650a3a693febfc9f1f4 2023-04-28 04:56:40,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 04:56:40,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=973 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/75e7f69a9bcc4cd196539d825680e914 2023-04-28 04:56:40,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=973 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/8bf2788e24844d01897079fe15f48aac 2023-04-28 04:56:40,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/A/590a4a927c944650a3a693febfc9f1f4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/590a4a927c944650a3a693febfc9f1f4 2023-04-28 04:56:40,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/590a4a927c944650a3a693febfc9f1f4, entries=150, sequenceid=973, filesize=25.4 K 2023-04-28 04:56:41,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/B/75e7f69a9bcc4cd196539d825680e914 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/75e7f69a9bcc4cd196539d825680e914 2023-04-28 04:56:41,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/75e7f69a9bcc4cd196539d825680e914, entries=150, sequenceid=973, filesize=11.9 K 2023-04-28 04:56:41,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/.tmp/C/8bf2788e24844d01897079fe15f48aac as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/8bf2788e24844d01897079fe15f48aac 2023-04-28 04:56:41,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/8bf2788e24844d01897079fe15f48aac, entries=150, sequenceid=973, filesize=11.9 K 2023-04-28 04:56:41,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 17eb93b75efa028513e70ba389775739 in 1707ms, sequenceid=973, compaction requested=false 2023-04-28 04:56:41,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5a265b5341e6488d9360e0871ea050dd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c9f2ba63cfa0453e9ba9e7d8a52bf071, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3280eae95e3b4f1f9745a77a8e56a787, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/8be5cec31be34b808772cbf669f5a07e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/e041720ef4ae4bfabf101c41d075ed2d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5234e3bee59b4bacb2e20048da4a5421, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/748a4f90d0b2466e9949f51978a837f0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c654e8ecc20e4659a1c98647e463e87a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/a0bf57e4abfb4e9e885b614fb70c761d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5db41099cbb442b78592e07c610af955, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/bf86a79956b0442580d590674b6e023c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/714ff7d599744095a60507a96eec1df7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3cb46d234949450fb1e78b35014464b0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/724a4d76af934b9696d9d615649197d9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3e21a1a18db94bf7b3a207e372fe17ba, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/dfd8a58ec7d641379972ba098e22c2b0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ec6e75da43bf46b3ae70967729417b93, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/7b29eadd084a43cd8206f65fefa28bfa, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c913d814b8e847f4928bd662b44d28f0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/01af7fd56ff34141aace722aad7e5cb8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/6918d024b3444bb297dbd97a450dd5a1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/011c94d5d28142758e732d97ab23d5c5] to archive 2023-04-28 04:56:41,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:56:41,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5a265b5341e6488d9360e0871ea050dd to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5a265b5341e6488d9360e0871ea050dd 2023-04-28 04:56:41,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c9f2ba63cfa0453e9ba9e7d8a52bf071 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c9f2ba63cfa0453e9ba9e7d8a52bf071 2023-04-28 04:56:41,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3280eae95e3b4f1f9745a77a8e56a787 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3280eae95e3b4f1f9745a77a8e56a787 2023-04-28 04:56:41,086 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/8be5cec31be34b808772cbf669f5a07e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/8be5cec31be34b808772cbf669f5a07e 2023-04-28 04:56:41,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/e041720ef4ae4bfabf101c41d075ed2d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/e041720ef4ae4bfabf101c41d075ed2d 2023-04-28 04:56:41,088 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5234e3bee59b4bacb2e20048da4a5421 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5234e3bee59b4bacb2e20048da4a5421 2023-04-28 04:56:41,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/748a4f90d0b2466e9949f51978a837f0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/748a4f90d0b2466e9949f51978a837f0 2023-04-28 04:56:41,091 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c654e8ecc20e4659a1c98647e463e87a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c654e8ecc20e4659a1c98647e463e87a 2023-04-28 04:56:41,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/a0bf57e4abfb4e9e885b614fb70c761d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/a0bf57e4abfb4e9e885b614fb70c761d 2023-04-28 04:56:41,093 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5db41099cbb442b78592e07c610af955 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/5db41099cbb442b78592e07c610af955 2023-04-28 04:56:41,094 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/bf86a79956b0442580d590674b6e023c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/bf86a79956b0442580d590674b6e023c 2023-04-28 04:56:41,095 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/714ff7d599744095a60507a96eec1df7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/714ff7d599744095a60507a96eec1df7 2023-04-28 04:56:41,096 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3cb46d234949450fb1e78b35014464b0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3cb46d234949450fb1e78b35014464b0 2023-04-28 04:56:41,097 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/724a4d76af934b9696d9d615649197d9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/724a4d76af934b9696d9d615649197d9 2023-04-28 04:56:41,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3e21a1a18db94bf7b3a207e372fe17ba to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/3e21a1a18db94bf7b3a207e372fe17ba 2023-04-28 04:56:41,100 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/dfd8a58ec7d641379972ba098e22c2b0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/dfd8a58ec7d641379972ba098e22c2b0 2023-04-28 04:56:41,101 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ec6e75da43bf46b3ae70967729417b93 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ec6e75da43bf46b3ae70967729417b93 2023-04-28 04:56:41,102 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/7b29eadd084a43cd8206f65fefa28bfa to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/7b29eadd084a43cd8206f65fefa28bfa 2023-04-28 04:56:41,103 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c913d814b8e847f4928bd662b44d28f0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/c913d814b8e847f4928bd662b44d28f0 2023-04-28 04:56:41,105 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/01af7fd56ff34141aace722aad7e5cb8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/01af7fd56ff34141aace722aad7e5cb8 2023-04-28 04:56:41,106 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/6918d024b3444bb297dbd97a450dd5a1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/6918d024b3444bb297dbd97a450dd5a1 2023-04-28 04:56:41,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/011c94d5d28142758e732d97ab23d5c5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/011c94d5d28142758e732d97ab23d5c5 2023-04-28 04:56:41,109 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/76a3137838da4416b604c88444174e59, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/2a52fa26c8104ac1a4c2da8f09805316, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/a5060ca895c940df9a90bcbd9c8baea4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d8e895b88c8b4ec18976685573e9701a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/514b501691ac478187e4c02169d63549, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/9810a73debce48bb850abbb6a004b048, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/867cf39946654123aa202d227b193f0f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/ad98e35f3c0943f0ba7d43ae663eefff, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/359da8f89c2e47e4bb68069c1317ec75, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d283ef9931a14438baf746b02cd084bf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/878faf95a008432cb97d5347e34be873, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07efbc44a60f46ddb11ff5e5378e9f21, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/7390c2fc91dd47259d94ca5054340d1d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/98a3ddf79e6849d2b6049771fc28381d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/791c9e2cbdb24e0e90cd0889ec882625, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/4c6fbe0c76a0476fb0c1afa1e1e2a163, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07b973e91361425db13480edc773b15d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d01dcf96d200449e811e16955ba674ed, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/460388449ac84963b8d1b78befae864f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/6c531b33488e4a2ca09a7edeb1a07993, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/973b74cf14314e06ac68dc32331222d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/5ec1724379584db283bc2a3e515af04b] to archive 2023-04-28 04:56:41,110 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:56:41,112 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/76a3137838da4416b604c88444174e59 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/76a3137838da4416b604c88444174e59 2023-04-28 04:56:41,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/2a52fa26c8104ac1a4c2da8f09805316 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/2a52fa26c8104ac1a4c2da8f09805316 2023-04-28 04:56:41,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/a5060ca895c940df9a90bcbd9c8baea4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/a5060ca895c940df9a90bcbd9c8baea4 2023-04-28 04:56:41,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d8e895b88c8b4ec18976685573e9701a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d8e895b88c8b4ec18976685573e9701a 2023-04-28 04:56:41,116 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/514b501691ac478187e4c02169d63549 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/514b501691ac478187e4c02169d63549 2023-04-28 04:56:41,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/9810a73debce48bb850abbb6a004b048 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/9810a73debce48bb850abbb6a004b048 2023-04-28 04:56:41,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/867cf39946654123aa202d227b193f0f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/867cf39946654123aa202d227b193f0f 2023-04-28 04:56:41,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/ad98e35f3c0943f0ba7d43ae663eefff to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/ad98e35f3c0943f0ba7d43ae663eefff 2023-04-28 04:56:41,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/359da8f89c2e47e4bb68069c1317ec75 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/359da8f89c2e47e4bb68069c1317ec75 2023-04-28 04:56:41,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d283ef9931a14438baf746b02cd084bf to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d283ef9931a14438baf746b02cd084bf 2023-04-28 04:56:41,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/878faf95a008432cb97d5347e34be873 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/878faf95a008432cb97d5347e34be873 2023-04-28 04:56:41,122 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07efbc44a60f46ddb11ff5e5378e9f21 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07efbc44a60f46ddb11ff5e5378e9f21 2023-04-28 04:56:41,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/7390c2fc91dd47259d94ca5054340d1d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/7390c2fc91dd47259d94ca5054340d1d 2023-04-28 04:56:41,124 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/98a3ddf79e6849d2b6049771fc28381d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/98a3ddf79e6849d2b6049771fc28381d 2023-04-28 04:56:41,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/791c9e2cbdb24e0e90cd0889ec882625 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/791c9e2cbdb24e0e90cd0889ec882625 2023-04-28 04:56:41,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/4c6fbe0c76a0476fb0c1afa1e1e2a163 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/4c6fbe0c76a0476fb0c1afa1e1e2a163 2023-04-28 04:56:41,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07b973e91361425db13480edc773b15d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/07b973e91361425db13480edc773b15d 2023-04-28 04:56:41,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d01dcf96d200449e811e16955ba674ed to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/d01dcf96d200449e811e16955ba674ed 2023-04-28 04:56:41,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/460388449ac84963b8d1b78befae864f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/460388449ac84963b8d1b78befae864f 2023-04-28 04:56:41,129 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/6c531b33488e4a2ca09a7edeb1a07993 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/6c531b33488e4a2ca09a7edeb1a07993 2023-04-28 04:56:41,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/973b74cf14314e06ac68dc32331222d3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/973b74cf14314e06ac68dc32331222d3 2023-04-28 04:56:41,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/5ec1724379584db283bc2a3e515af04b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/5ec1724379584db283bc2a3e515af04b 2023-04-28 04:56:41,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/abb2e76894f94e5fb5ed116b1852e1fc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/388276f526fd44d7bfa64be636bd5366, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4d5c9db1b1664932b3cfce19fc268fe1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/3f0164fae78642d9a344fc8e76d092b7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/294c5e0c430c45418083119c958afeff, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/94ed88a84ad64250b512aeb618052a07, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/516be6a7aa9a402883e3803dd3301f1a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/670085c45ed34f52b7843cca0e9020ef, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/573cc5737ce14e808b1283bfe43e051f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/b072903ee5d244cda63c9f20366dc0a2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/a509815e851c428abde904002257431b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/f06a2141216a4d009fed17ebd300a7c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4f2db6d2ed2f4cc89d5cb58b42edb624, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/752e62b406be40abab43e80b75a06936, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/e14b742bf4b14f0baf3cc5486ac27b7a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/517d2f53fbff45a79003cae79600c9a1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/2564f094634f4653b0e3bf0722540dbc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/1d6c2ca22581450d8193c36930dfa913, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/04c81ab1b8994622ad3dd99e2f7d7703, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/ca2c4d2cc5274f2181489e4857da052e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/89140c4d5958494b8db2acc7ba4d2ed9] to archive 2023-04-28 04:56:41,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:56:41,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/abb2e76894f94e5fb5ed116b1852e1fc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/abb2e76894f94e5fb5ed116b1852e1fc 2023-04-28 04:56:41,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/388276f526fd44d7bfa64be636bd5366 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/388276f526fd44d7bfa64be636bd5366 2023-04-28 04:56:41,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4d5c9db1b1664932b3cfce19fc268fe1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4d5c9db1b1664932b3cfce19fc268fe1 2023-04-28 04:56:41,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/3f0164fae78642d9a344fc8e76d092b7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/3f0164fae78642d9a344fc8e76d092b7 2023-04-28 04:56:41,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/294c5e0c430c45418083119c958afeff to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/294c5e0c430c45418083119c958afeff 2023-04-28 04:56:41,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/94ed88a84ad64250b512aeb618052a07 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/94ed88a84ad64250b512aeb618052a07 2023-04-28 04:56:41,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/516be6a7aa9a402883e3803dd3301f1a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/516be6a7aa9a402883e3803dd3301f1a 2023-04-28 04:56:41,140 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/670085c45ed34f52b7843cca0e9020ef to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/670085c45ed34f52b7843cca0e9020ef 2023-04-28 04:56:41,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/573cc5737ce14e808b1283bfe43e051f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/573cc5737ce14e808b1283bfe43e051f 2023-04-28 04:56:41,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/b072903ee5d244cda63c9f20366dc0a2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/b072903ee5d244cda63c9f20366dc0a2 2023-04-28 04:56:41,144 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/a509815e851c428abde904002257431b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/a509815e851c428abde904002257431b 2023-04-28 04:56:41,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/056bbd3cbcaf4ddeb015fd3e10bcb5e2 2023-04-28 04:56:41,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/f06a2141216a4d009fed17ebd300a7c0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/f06a2141216a4d009fed17ebd300a7c0 2023-04-28 04:56:41,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4f2db6d2ed2f4cc89d5cb58b42edb624 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/4f2db6d2ed2f4cc89d5cb58b42edb624 2023-04-28 04:56:41,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/752e62b406be40abab43e80b75a06936 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/752e62b406be40abab43e80b75a06936 2023-04-28 04:56:41,151 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/e14b742bf4b14f0baf3cc5486ac27b7a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/e14b742bf4b14f0baf3cc5486ac27b7a 2023-04-28 04:56:41,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/517d2f53fbff45a79003cae79600c9a1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/517d2f53fbff45a79003cae79600c9a1 2023-04-28 04:56:41,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/2564f094634f4653b0e3bf0722540dbc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/2564f094634f4653b0e3bf0722540dbc 2023-04-28 04:56:41,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/1d6c2ca22581450d8193c36930dfa913 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/1d6c2ca22581450d8193c36930dfa913 2023-04-28 04:56:41,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/04c81ab1b8994622ad3dd99e2f7d7703 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/04c81ab1b8994622ad3dd99e2f7d7703 2023-04-28 04:56:41,159 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/ca2c4d2cc5274f2181489e4857da052e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/ca2c4d2cc5274f2181489e4857da052e 2023-04-28 04:56:41,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/89140c4d5958494b8db2acc7ba4d2ed9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/89140c4d5958494b8db2acc7ba4d2ed9 2023-04-28 04:56:41,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/recovered.edits/976.seqid, newMaxSeqId=976, maxSeqId=4 2023-04-28 04:56:41,169 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739. 2023-04-28 04:56:41,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 17eb93b75efa028513e70ba389775739: 2023-04-28 04:56:41,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed 17eb93b75efa028513e70ba389775739 2023-04-28 04:56:41,172 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=17eb93b75efa028513e70ba389775739, regionState=CLOSED 2023-04-28 04:56:41,172 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657801171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657801171"}]},"ts":"1682657801171"} 2023-04-28 04:56:41,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-28 04:56:41,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 17eb93b75efa028513e70ba389775739, server=jenkins-hbase12.apache.org,37663,1682657733313 in 2.0210 sec 2023-04-28 04:56:41,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-28 04:56:41,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=17eb93b75efa028513e70ba389775739, UNASSIGN in 2.0250 sec 2023-04-28 04:56:41,177 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657801177"}]},"ts":"1682657801177"} 2023-04-28 04:56:41,178 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 04:56:41,201 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 04:56:41,203 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1040 sec 2023-04-28 04:56:41,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 04:56:41,208 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-28 04:56:41,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$5(2228): Client=jenkins//136.243.104.168 delete TestAcidGuarantees 2023-04-28 04:56:41,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,212 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 04:56:41,213 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,218 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:41,220 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/recovered.edits] 2023-04-28 04:56:41,224 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/590a4a927c944650a3a693febfc9f1f4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/590a4a927c944650a3a693febfc9f1f4 2023-04-28 04:56:41,226 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ba26277961f44a21bdc4ac4b15574eab to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/A/ba26277961f44a21bdc4ac4b15574eab 2023-04-28 04:56:41,229 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/0013147da5d443dcaefd4d161e040d8b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/0013147da5d443dcaefd4d161e040d8b 2023-04-28 04:56:41,231 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/75e7f69a9bcc4cd196539d825680e914 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/B/75e7f69a9bcc4cd196539d825680e914 2023-04-28 04:56:41,235 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/340bb7213e434dad925f4b5b16ce98d3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/340bb7213e434dad925f4b5b16ce98d3 2023-04-28 04:56:41,237 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/8bf2788e24844d01897079fe15f48aac to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/C/8bf2788e24844d01897079fe15f48aac 2023-04-28 04:56:41,247 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/recovered.edits/976.seqid to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739/recovered.edits/976.seqid 2023-04-28 04:56:41,248 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/17eb93b75efa028513e70ba389775739 2023-04-28 04:56:41,248 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:56:41,248 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 04:56:41,249 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 04:56:41,254 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042806837e4c133e49198018f6a0c9471f4f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042806837e4c133e49198018f6a0c9471f4f 2023-04-28 04:56:41,257 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428173bf49a81db40848a13f05c559ef4a2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428173bf49a81db40848a13f05c559ef4a2 2023-04-28 04:56:41,258 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281b6b6c963152452995f232f0b9ed030e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281b6b6c963152452995f232f0b9ed030e 2023-04-28 04:56:41,259 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428336bce030c574de9827a6ef1153592e7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428336bce030c574de9827a6ef1153592e7 2023-04-28 04:56:41,264 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042844494754bbe84df7b2a09bcd79f2e7fe to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042844494754bbe84df7b2a09bcd79f2e7fe 2023-04-28 04:56:41,266 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428506df3da70454765801b3de0db24808b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428506df3da70454765801b3de0db24808b 2023-04-28 04:56:41,269 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042856eb16ada0154f91886c50853151e208 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042856eb16ada0154f91886c50853151e208 2023-04-28 04:56:41,271 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428597722ff9b054577bdcde346f35b11bd to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428597722ff9b054577bdcde346f35b11bd 2023-04-28 04:56:41,273 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428766a688d9dda47b1b5109935afd13324 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428766a688d9dda47b1b5109935afd13324 2023-04-28 04:56:41,276 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304288e55a7d5ede54e5c894c32b4b8ff983e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304288e55a7d5ede54e5c894c32b4b8ff983e 2023-04-28 04:56:41,277 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a26f299d790a4d12a1a443a49be99496 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a26f299d790a4d12a1a443a49be99496 2023-04-28 04:56:41,279 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ae21111f7e8c447ba674c9bb963ea93d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ae21111f7e8c447ba674c9bb963ea93d 2023-04-28 04:56:41,280 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428be0e10d1114545a4bbc5190e8b8a9001 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428be0e10d1114545a4bbc5190e8b8a9001 2023-04-28 04:56:41,281 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c5387b4364734dfe99047c7e9765fa19 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c5387b4364734dfe99047c7e9765fa19 2023-04-28 04:56:41,283 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c8477a5ccc3f48bfbd41ebb51020f662 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c8477a5ccc3f48bfbd41ebb51020f662 2023-04-28 04:56:41,285 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cbbde4f8a00b4d0180fa3108a2986f14 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cbbde4f8a00b4d0180fa3108a2986f14 2023-04-28 04:56:41,287 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d1eef46ef1e84ef88c13627bde72a1a0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d1eef46ef1e84ef88c13627bde72a1a0 2023-04-28 04:56:41,290 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428db9247c146e14202820a782f7bcc1745 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428db9247c146e14202820a782f7bcc1745 2023-04-28 04:56:41,294 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e04b7cfa414d4ec6bc103427da540edb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e04b7cfa414d4ec6bc103427da540edb 2023-04-28 04:56:41,295 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 04:56:41,299 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,303 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 04:56:41,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 04:56:41,318 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 04:56:41,319 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,319 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 04:56:41,319 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682657801319"}]},"ts":"9223372036854775807"} 2023-04-28 04:56:41,322 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 04:56:41,322 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 17eb93b75efa028513e70ba389775739, NAME => 'TestAcidGuarantees,,1682657765365.17eb93b75efa028513e70ba389775739.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 04:56:41,322 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 04:56:41,322 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682657801322"}]},"ts":"9223372036854775807"} 2023-04-28 04:56:41,324 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 04:56:41,371 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 162 msec 2023-04-28 04:56:41,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 04:56:41,516 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-28 04:56:41,530 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 233), OpenFileDescriptor=539 (was 548), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=395 (was 301) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=9959 (was 9949) - AvailableMemoryMB LEAK? - 2023-04-28 04:56:41,540 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=395, ProcessCount=171, AvailableMemoryMB=9958 2023-04-28 04:56:41,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:56:41,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$4(2112): Client=jenkins//136.243.104.168 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:56:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 04:56:41,544 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:56:41,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(697): Client=jenkins//136.243.104.168 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-28 04:56:41,545 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:56:41,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 04:56:41,546 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:41,547 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 empty. 2023-04-28 04:56:41,547 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:41,547 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:56:41,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 04:56:41,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 04:56:41,958 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:56:41,961 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bf0b022e81bb29fa4ec0f9acedc0e278, NAME => 'TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:56:42,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 04:56:42,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:56:42,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bf0b022e81bb29fa4ec0f9acedc0e278, disabling compactions & flushes 2023-04-28 04:56:42,370 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. after waiting 0 ms 2023-04-28 04:56:42,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,370 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:42,372 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:56:42,373 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657802373"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657802373"}]},"ts":"1682657802373"} 2023-04-28 04:56:42,374 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:56:42,374 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:56:42,374 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657802374"}]},"ts":"1682657802374"} 2023-04-28 04:56:42,375 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 04:56:42,401 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, ASSIGN}] 2023-04-28 04:56:42,403 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, ASSIGN 2023-04-28 04:56:42,404 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:56:42,555 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=bf0b022e81bb29fa4ec0f9acedc0e278, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:42,555 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657802555"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657802555"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657802555"}]},"ts":"1682657802555"} 2023-04-28 04:56:42,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:56:42,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 04:56:42,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bf0b022e81bb29fa4ec0f9acedc0e278, NAME => 'TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:56:42,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:56:42,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,717 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,721 DEBUG [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A 2023-04-28 04:56:42,721 DEBUG [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A 2023-04-28 04:56:42,721 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:42,721 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf0b022e81bb29fa4ec0f9acedc0e278 columnFamilyName A 2023-04-28 04:56:42,722 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.HStore(310): Store=bf0b022e81bb29fa4ec0f9acedc0e278/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:42,722 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,724 DEBUG [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B 2023-04-28 04:56:42,724 DEBUG [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B 2023-04-28 04:56:42,724 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:42,724 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf0b022e81bb29fa4ec0f9acedc0e278 columnFamilyName B 2023-04-28 04:56:42,725 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.HStore(310): Store=bf0b022e81bb29fa4ec0f9acedc0e278/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:42,725 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,726 DEBUG [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C 2023-04-28 04:56:42,726 DEBUG [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C 2023-04-28 04:56:42,726 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:56:42,727 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf0b022e81bb29fa4ec0f9acedc0e278 columnFamilyName C 2023-04-28 04:56:42,727 INFO [StoreOpener-bf0b022e81bb29fa4ec0f9acedc0e278-1] regionserver.HStore(310): Store=bf0b022e81bb29fa4ec0f9acedc0e278/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:56:42,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,730 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:56:42,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:42,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:56:42,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened bf0b022e81bb29fa4ec0f9acedc0e278; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62224824, jitterRate=-0.07277786731719971}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:56:42,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:42,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., pid=30, masterSystemTime=1682657802708 2023-04-28 04:56:42,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:42,736 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=bf0b022e81bb29fa4ec0f9acedc0e278, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:42,737 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657802736"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657802736"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657802736"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657802736"}]},"ts":"1682657802736"} 2023-04-28 04:56:42,739 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-28 04:56:42,739 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 in 180 msec 2023-04-28 04:56:42,740 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-28 04:56:42,740 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, ASSIGN in 338 msec 2023-04-28 04:56:42,740 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:56:42,740 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657802740"}]},"ts":"1682657802740"} 2023-04-28 04:56:42,741 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 04:56:42,760 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:56:42,761 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2180 sec 2023-04-28 04:56:43,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 04:56:43,655 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-28 04:56:43,656 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x12394d27 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,676 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7187fcf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,679 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,680 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58466, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,681 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:56:43,685 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:48534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:56:43,686 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x29a548ac to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,706 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@712d78ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,707 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x43184c57 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,728 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@407b9cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,728 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x7e3efec6 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,749 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f456fa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,750 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x6b7f716c to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,770 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62be7a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,771 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x6ade356f to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,791 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abacd49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,792 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x57d25552 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,812 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bae1a64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,812 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x13695730 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,833 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5654f8ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,834 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x0993cc7d to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,844 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@509adaa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,845 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x69b9b3fd to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,906 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e5e7a93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,906 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x1656b283 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:56:43,928 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3254ddef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:56:43,935 DEBUG [hconnection-0x2b845ad2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,941 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,943 DEBUG [hconnection-0x691670ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,944 DEBUG [hconnection-0x3862ded0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,950 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:43,980 DEBUG [hconnection-0x5854f65d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,985 DEBUG [hconnection-0x5fa9fe32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,985 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58494, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,986 DEBUG [hconnection-0x6000d087-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,987 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,987 DEBUG [hconnection-0x3bcc501c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,993 DEBUG [hconnection-0x2dd9602-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,995 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,995 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,993 DEBUG [hconnection-0x558c4dd5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,995 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:43,995 DEBUG [hconnection-0x5cc2f627-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:56:43,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:43,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:43,997 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:43,999 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:43,999 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:43,999 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:44,000 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:44,000 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:44,000 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,002 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:58564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:56:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:44,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:56:44,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:44,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:44,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:44,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:44,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:44,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:44,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657864026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657864028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657804023, max timestamp=1682657804031Num uniques 859; 2023-04-28 04:56:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 04:56:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657804023, max timestamp=1682657804028Num uniques 859; 2023-04-28 04:56:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:44,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657804023, max timestamp=1682657804028Num uniques 859; 2023-04-28 04:56:44,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 04:56:44,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:44,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:44,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:44,048 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,048 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:44,048 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:44,048 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:44,048 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:44,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657864060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:44,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657864061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657864062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,065 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f2825a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:44,067 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:44,067 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,075 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:44,075 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:44,075 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,075 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:44,075 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:44,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:44,085 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:44,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:44,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:44,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:44,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:44,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:44,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:44,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,096 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,097 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:44,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:44,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,107 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:44,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:44,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,107 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:44,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657864133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657864133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657864164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657864166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657864166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,207 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:44,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:44,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:44,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:44,210 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:44,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:44,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:44,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,222 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:44,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:44,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:44,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:44,223 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,223 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:44,223 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:44,223 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:44,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:44,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:44,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:44,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,232 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,233 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,233 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:44,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5368c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:44,234 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:44,234 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:44,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:44,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:44,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:44,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:44,253 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,254 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,254 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:44,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:44,256 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:44,256 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:44,256 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:44,256 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:44,256 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:44,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,265 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,265 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:44,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:44,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:44,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,275 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:44,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,275 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:44,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657864337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657864338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657864369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657864370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657864370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,375 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:44,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:44,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:44,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:44,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:44,378 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:44,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:44,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:44,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:44,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:44,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:44,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:44,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:44,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:44,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:44,401 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f44a29d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:44,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:44,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:44,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:44,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:44,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:44,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:44,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:44,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:44,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:44,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:44,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:44,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:44,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:44,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/525a51f1006c4792a642b87e7971a7c1 2023-04-28 04:56:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,435 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:44,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:44,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,444 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:44,444 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,444 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:44,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,544 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:44,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:44,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:44,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:44,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:44,551 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:44,551 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,551 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:44,551 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:44,551 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,551 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,559 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,559 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,559 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,559 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:44,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:44,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:44,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:44,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:44,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:44,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:44,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:44,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:44,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,561 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:44,561 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:44,561 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,561 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:44,561 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:44,601 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:44,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5619ef7e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:44,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:44,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:44,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:44,622 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,622 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:44,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:44,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:44,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:44,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:44,640 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:44,640 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:44,640 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:44,640 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:44,640 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:44,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657864639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657864641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,643 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:44,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:44,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:44,654 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:44,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,654 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657864671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657864672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:44,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657864673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,754 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:44,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:44,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:44,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:44,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:44,757 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:44,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:44,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:44,758 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,758 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:44,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:44,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:44,771 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,771 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:44,771 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:44,771 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:44,771 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:44,780 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,780 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,780 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,780 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:44,782 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301f57b5[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:44,782 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:44,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:44,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:44,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:44,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:44,801 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:44,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:44,803 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:44,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:44,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:44,803 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:44,803 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:44,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:44,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,822 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,822 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:44,822 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:44,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5d683fa5ea334109a172a56203e6392a 2023-04-28 04:56:44,922 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:44,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:44,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:44,925 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:44,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:44,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:44,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,938 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,938 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,938 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,938 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:44,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:44,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:44,939 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,939 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:44,939 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:44,939 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:44,939 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:44,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,948 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:44,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc1666c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:44,950 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:44,950 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,959 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:44,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:44,959 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,959 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:44,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:44,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:44,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:44,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:44,971 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:44,971 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:44,971 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:44,971 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:44,971 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:44,980 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,980 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:44,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,980 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:44,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,981 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:44,990 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:44,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:44,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:44,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:44,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:44,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:44,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:44,991 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:44,991 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,991 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:44,991 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:44,991 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:44,991 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:44,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:45,091 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:45,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:45,094 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:45,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:45,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:45,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:45,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:45,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:45,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:45,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:45,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:45,107 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:45,107 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:45,107 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:45,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:45,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:45,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:45,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:45,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:45,117 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,117 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:45,117 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,117 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:45,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:45,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b7f43[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:45,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:45,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:45,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:45,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:45,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:45,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657865143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657865146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:45,164 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:45,164 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:45,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:45,166 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:45,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:45,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:45,166 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:45,166 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:45,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:45,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657865175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:45,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657865178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:45,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657865178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,180 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,180 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,180 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:45,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:45,181 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:45,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,190 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:45,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:45,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:45,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:45,191 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:45,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:45,191 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:45,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:45,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:45,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/e0f76405a6714bec92445687eb837d18 2023-04-28 04:56:45,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/525a51f1006c4792a642b87e7971a7c1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/525a51f1006c4792a642b87e7971a7c1 2023-04-28 04:56:45,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/525a51f1006c4792a642b87e7971a7c1, entries=150, sequenceid=12, filesize=11.6 K 2023-04-28 04:56:45,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5d683fa5ea334109a172a56203e6392a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d683fa5ea334109a172a56203e6392a 2023-04-28 04:56:45,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d683fa5ea334109a172a56203e6392a, entries=150, sequenceid=12, filesize=11.6 K 2023-04-28 04:56:45,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/e0f76405a6714bec92445687eb837d18 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/e0f76405a6714bec92445687eb837d18 2023-04-28 04:56:45,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/e0f76405a6714bec92445687eb837d18, entries=150, sequenceid=12, filesize=11.6 K 2023-04-28 04:56:45,272 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129720 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1261ms, sequenceid=12, compaction requested=false 2023-04-28 04:56:45,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:45,291 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:45,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:45,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:45,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:45,294 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:45,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:45,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:45,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:45,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,306 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:45,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:45,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,307 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:45,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:45,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:45,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:45,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:45,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:45,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:45,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:45,308 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:45,308 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:45,323 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/efd97a439a694a629b946cd5726dc83b 2023-04-28 04:56:45,333 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/cb8c078b26c94b279e627470264c430c 2023-04-28 04:56:45,768 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/660d3eb790354e24b3330e310acb7a68 2023-04-28 04:56:45,772 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/efd97a439a694a629b946cd5726dc83b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/efd97a439a694a629b946cd5726dc83b 2023-04-28 04:56:45,776 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/efd97a439a694a629b946cd5726dc83b, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 04:56:45,776 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/cb8c078b26c94b279e627470264c430c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/cb8c078b26c94b279e627470264c430c 2023-04-28 04:56:45,783 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/cb8c078b26c94b279e627470264c430c, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 04:56:45,784 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/660d3eb790354e24b3330e310acb7a68 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/660d3eb790354e24b3330e310acb7a68 2023-04-28 04:56:45,792 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/660d3eb790354e24b3330e310acb7a68, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 04:56:45,794 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-28 04:56:45,794 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-28 04:56:45,795 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=0 B/0 for bf0b022e81bb29fa4ec0f9acedc0e278 in 487ms, sequenceid=48, compaction requested=false 2023-04-28 04:56:45,795 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:45,795 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:45,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:45,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:45,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:45,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:45,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:45,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:45,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:45,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c700293[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:45,845 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:45,845 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,854 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:45,854 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,854 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:45,854 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:45,854 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:45,864 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:45,864 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,864 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:45,864 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:45,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:45,866 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:45,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:45,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:45,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:45,867 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:45,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:45,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,875 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:45,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,876 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:45,877 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:45,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:45,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:45,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:45,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:45,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:45,885 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:45,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:45,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:45,885 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:45,886 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,886 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:45,886 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:45,986 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:45,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:45,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:45,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:45,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:45,989 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:45,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:45,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:45,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:45,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:45,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:46,002 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:46,002 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:46,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:46,002 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:46,002 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:46,003 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,003 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:46,003 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:46,003 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:46,003 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:46,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56984c57[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:46,013 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:46,013 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,022 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,022 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:46,022 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:46,022 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:46,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:46,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:46,032 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:46,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:46,034 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:46,035 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:46,035 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:46,035 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:46,035 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:46,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,135 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:46,136 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,136 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:46,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:46,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:56:46,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:46,160 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:46,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:46,160 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:46,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:46,160 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:46,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/6158b64652b04f3c917d6a8c646d8f58 2023-04-28 04:56:46,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657866187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657806184, max timestamp=1682657806189Num uniques 859; 2023-04-28 04:56:46,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:46,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657806184, max timestamp=1682657806189Num uniques 859; 2023-04-28 04:56:46,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:46,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657806184, max timestamp=1682657806189Num uniques 859; 2023-04-28 04:56:46,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 04:56:46,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657866212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657866214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657866214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657866215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,236 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:46,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:46,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:46,240 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:46,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:46,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:46,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:46,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:46,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:46,265 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:46,265 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:46,266 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,266 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:46,266 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:46,266 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:46,266 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:46,275 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:46,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@427ea2ee[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:46,277 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:46,278 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,285 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,285 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:46,285 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:46,285 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:46,285 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,285 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:46,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,296 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:46,296 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,296 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:46,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657866296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:46,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:46,298 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:46,298 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:46,298 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:46,298 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:46,298 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657866317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657866317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657866317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657866318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,333 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,333 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,333 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,333 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,335 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:46,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,349 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,349 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,349 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:46,349 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,449 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:46,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:46,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:46,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:46,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:46,452 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:46,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:46,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:46,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,464 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:46,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:46,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:46,466 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,466 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:46,466 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:46,466 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:46,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:46,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:46,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:46,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c51ac85[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:46,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,485 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:46,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:46,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:46,485 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,485 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:46,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:46,496 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:46,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:46,497 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:46,498 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:46,498 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:46,498 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:46,498 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:46,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657866498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:46,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:46,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,517 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,517 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:46,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657866520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657866521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657866521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657866522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,622 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:46,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:46,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:46,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:46,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:46,626 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:46,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:46,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:46,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,626 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/7ba61777016e42a69438131effb16c12 2023-04-28 04:56:46,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/48a967cc51fd40acbb22ae67e2d43e5e 2023-04-28 04:56:46,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/6158b64652b04f3c917d6a8c646d8f58 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6158b64652b04f3c917d6a8c646d8f58 2023-04-28 04:56:46,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6158b64652b04f3c917d6a8c646d8f58, entries=150, sequenceid=60, filesize=11.6 K 2023-04-28 04:56:46,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:46,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/7ba61777016e42a69438131effb16c12 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/7ba61777016e42a69438131effb16c12 2023-04-28 04:56:46,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:46,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:46,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:46,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:46,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:46,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:46,666 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,667 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:46,667 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:46,667 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,667 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:46,667 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:46,667 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,667 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:46,667 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:46,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/7ba61777016e42a69438131effb16c12, entries=150, sequenceid=60, filesize=11.6 K 2023-04-28 04:56:46,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/48a967cc51fd40acbb22ae67e2d43e5e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/48a967cc51fd40acbb22ae67e2d43e5e 2023-04-28 04:56:46,676 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:46,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:46,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294106[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:46,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/48a967cc51fd40acbb22ae67e2d43e5e, entries=150, sequenceid=60, filesize=11.6 K 2023-04-28 04:56:46,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for bf0b022e81bb29fa4ec0f9acedc0e278 in 521ms, sequenceid=60, compaction requested=true 2023-04-28 04:56:46,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:46,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:46,680 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:46,680 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:46,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:46,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:46,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,686 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:46,686 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:46,686 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:56:46,686 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:46,686 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:46,686 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,686 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:46,686 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/525a51f1006c4792a642b87e7971a7c1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/efd97a439a694a629b946cd5726dc83b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6158b64652b04f3c917d6a8c646d8f58] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=34.8 K 2023-04-28 04:56:46,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:46,686 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:56:46,686 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,687 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d683fa5ea334109a172a56203e6392a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/cb8c078b26c94b279e627470264c430c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/7ba61777016e42a69438131effb16c12] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=34.8 K 2023-04-28 04:56:46,687 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 525a51f1006c4792a642b87e7971a7c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682657804005 2023-04-28 04:56:46,687 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5d683fa5ea334109a172a56203e6392a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682657804005 2023-04-28 04:56:46,687 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting efd97a439a694a629b946cd5726dc83b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682657804055 2023-04-28 04:56:46,687 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting cb8c078b26c94b279e627470264c430c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682657804055 2023-04-28 04:56:46,688 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6158b64652b04f3c917d6a8c646d8f58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682657806158 2023-04-28 04:56:46,688 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7ba61777016e42a69438131effb16c12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682657806158 2023-04-28 04:56:46,693 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:46,694 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#182 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:46,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:46,696 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:46,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:46,698 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:46,698 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:46,698 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:46,698 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:46,698 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:46,704 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:56:46,706 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,707 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,713 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,727 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:46,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:46,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,728 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:46,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:46,728 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:46,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,736 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/c0fd289d1d184e079137a6129373dcd9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c0fd289d1d184e079137a6129373dcd9 2023-04-28 04:56:46,741 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into c0fd289d1d184e079137a6129373dcd9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:46,741 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:46,741 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=13, startTime=1682657806680; duration=0sec 2023-04-28 04:56:46,741 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:46,742 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:46,742 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:46,743 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:56:46,743 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,743 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/e0f76405a6714bec92445687eb837d18, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/660d3eb790354e24b3330e310acb7a68, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/48a967cc51fd40acbb22ae67e2d43e5e] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=34.8 K 2023-04-28 04:56:46,743 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting e0f76405a6714bec92445687eb837d18, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682657804005 2023-04-28 04:56:46,743 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 660d3eb790354e24b3330e310acb7a68, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682657804055 2023-04-28 04:56:46,743 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 48a967cc51fd40acbb22ae67e2d43e5e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682657806158 2023-04-28 04:56:46,765 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#184 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:46,776 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/b1af301275a44886ad0a06b32d9e3d48 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1af301275a44886ad0a06b32d9e3d48 2023-04-28 04:56:46,782 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into b1af301275a44886ad0a06b32d9e3d48(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:46,782 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:46,782 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=13, startTime=1682657806680; duration=0sec 2023-04-28 04:56:46,783 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:46,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:46,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-04-28 04:56:46,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:46,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:46,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:46,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:46,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:46,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:46,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/d5b497cd3d9c455db808e9308669936c 2023-04-28 04:56:46,828 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:46,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657866826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:46,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:46,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657866827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657866828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657866828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657866829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:46,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:46,836 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:46,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:46,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:46,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,837 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:46,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:46,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:46,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:46,857 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:46,857 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:46,857 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:46,860 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,860 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:46,860 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:46,860 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:46,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:46,864 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,864 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,864 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,864 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:46,865 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:46,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c75fa0[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:46,866 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,874 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:46,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:46,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:46,875 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,875 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:46,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:46,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:46,887 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:46,887 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:46,887 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:46,887 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:46,887 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,896 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,897 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:46,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:46,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:46,934 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:46,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:46,934 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,934 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:46,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,935 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:46,935 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:46,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:46,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657866935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657866935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657866935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:46,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:46,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657866936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,035 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:47,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:47,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:47,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:47,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:47,038 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:47,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:47,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:47,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,053 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:47,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:47,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:47,055 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,055 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:47,055 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:47,055 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:47,055 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:47,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:47,065 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:47,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2399f604[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:47,067 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:47,067 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,075 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:47,075 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:47,075 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,075 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:47,075 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:47,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:47,085 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:47,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:47,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:47,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:47,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:47,087 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:47,087 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:47,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,096 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,097 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:47,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,106 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,107 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,107 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,120 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/3a6d9dd852fa413aaf97eb3f23bea0b9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3a6d9dd852fa413aaf97eb3f23bea0b9 2023-04-28 04:56:47,124 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into 3a6d9dd852fa413aaf97eb3f23bea0b9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:47,124 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:47,124 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=13, startTime=1682657806680; duration=0sec 2023-04-28 04:56:47,124 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:47,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657867138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657867138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657867139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657867140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,207 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:47,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:47,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:47,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:47,209 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:47,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:47,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:47,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,222 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:47,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:47,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:47,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:47,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,223 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:47,224 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,224 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:47,224 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:47,224 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:47,224 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:47,232 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,233 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,233 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,233 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:47,234 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:47,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32bd5376[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:47,234 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:47,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:47,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:47,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:47,253 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,254 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:47,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:47,255 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:47,255 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:47,255 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:47,255 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:47,255 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:47,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/03152deba7e74dc18ee2aafcfdd2fa41 2023-04-28 04:56:47,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,265 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,267 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:47,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,275 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:47,275 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657867334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,375 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:47,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:47,378 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:47,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:47,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:47,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:47,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:47,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:47,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:47,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:47,397 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,397 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:47,397 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:47,397 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:47,397 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:47,401 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:47,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78561556[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:47,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:47,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:47,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:47,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:47,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:47,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:47,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:47,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:47,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:47,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:47,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:47,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:47,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:47,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,435 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657867441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657867441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657867442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:47,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657867444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,471 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,471 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,471 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,471 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:47,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:47,471 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,472 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:47,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:47,472 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:47,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,572 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:47,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:47,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:47,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:47,575 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:47,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:47,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:47,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,575 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,591 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,591 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:47,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:47,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:47,592 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,592 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:47,592 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:47,592 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:47,592 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:47,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,601 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,602 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,602 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:47,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:47,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56c27fa3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:47,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:47,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:47,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:47,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:47,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:47,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:47,624 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:47,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:47,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:47,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:47,624 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:47,633 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,634 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,643 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:47,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:47,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,643 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:47,643 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:47,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:47,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/5dc4e995f26e4e71a3c092da99d35384 2023-04-28 04:56:47,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/d5b497cd3d9c455db808e9308669936c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/d5b497cd3d9c455db808e9308669936c 2023-04-28 04:56:47,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/d5b497cd3d9c455db808e9308669936c, entries=200, sequenceid=99, filesize=13.9 K 2023-04-28 04:56:47,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/03152deba7e74dc18ee2aafcfdd2fa41 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/03152deba7e74dc18ee2aafcfdd2fa41 2023-04-28 04:56:47,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/03152deba7e74dc18ee2aafcfdd2fa41, entries=150, sequenceid=99, filesize=11.6 K 2023-04-28 04:56:47,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/5dc4e995f26e4e71a3c092da99d35384 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/5dc4e995f26e4e71a3c092da99d35384 2023-04-28 04:56:47,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/5dc4e995f26e4e71a3c092da99d35384, entries=150, sequenceid=99, filesize=11.6 K 2023-04-28 04:56:47,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=53.67 KB/54960 for bf0b022e81bb29fa4ec0f9acedc0e278 in 889ms, sequenceid=99, compaction requested=false 2023-04-28 04:56:47,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:47,744 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:47,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:47,747 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:47,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:47,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:47,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,759 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:47,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:47,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:47,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:47,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:47,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:47,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:47,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:47,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:47,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:47,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:47,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:47,760 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:47,761 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:47,761 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:47,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:47,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:47,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657807973, max timestamp=1682657807978Num uniques 859; 2023-04-28 04:56:47,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:47,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657807973, max timestamp=1682657807978Num uniques 859; 2023-04-28 04:56:47,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:47,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657807973, max timestamp=1682657807979Num uniques 859; 2023-04-28 04:56:47,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:56:48,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657868006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657868012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657868012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657868017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657868118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657868118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657868118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657868122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,196 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/55ee5f3d0d3948928465c55df69a2495 2023-04-28 04:56:48,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657868323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657868324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657868324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657868326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657868342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657868628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657868629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657868629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:48,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657868630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:48,642 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/41f8ca860f164070aed3e1ea42834907 2023-04-28 04:56:49,053 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/1ea0fa0d0f7444cd94e480784766eb62 2023-04-28 04:56:49,058 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/55ee5f3d0d3948928465c55df69a2495 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/55ee5f3d0d3948928465c55df69a2495 2023-04-28 04:56:49,074 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/55ee5f3d0d3948928465c55df69a2495, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:56:49,075 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/41f8ca860f164070aed3e1ea42834907 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f8ca860f164070aed3e1ea42834907 2023-04-28 04:56:49,102 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f8ca860f164070aed3e1ea42834907, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:56:49,103 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/1ea0fa0d0f7444cd94e480784766eb62 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/1ea0fa0d0f7444cd94e480784766eb62 2023-04-28 04:56:49,110 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/1ea0fa0d0f7444cd94e480784766eb62, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:56:49,111 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1350ms, sequenceid=111, compaction requested=true 2023-04-28 04:56:49,111 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:49,111 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,111 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:49,111 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:49,111 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,111 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:49,111 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:49,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:49,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-28 04:56:49,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:49,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:49,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:49,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:49,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:49,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:49,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657869147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657869148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657869148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657869151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:49,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a879aa9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:49,203 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:49,203 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:49,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:49,212 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,212 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:49,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:49,222 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,222 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:49,222 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,222 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:49,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:49,225 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:49,225 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:49,225 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:49,225 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:49,225 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:49,232 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,232 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,235 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,243 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:49,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:49,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,243 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:49,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,243 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:49,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657869256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657869256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657869256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657869256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,344 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:49,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:49,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:49,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:49,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:49,347 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:49,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:49,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:49,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:49,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:49,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:49,370 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:49,370 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,370 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:49,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:49,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:49,372 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,372 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:49,372 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:49,372 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:49,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:49,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,372 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:49,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:49,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:49,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f9486c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:49,383 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:49,383 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:49,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:49,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:49,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:49,402 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,402 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,402 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,402 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:49,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:49,405 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:49,405 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:49,405 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:49,405 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:49,405 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:49,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:49,423 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,423 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:49,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:49,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,423 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,424 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:49,424 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:49,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657869460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657869460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657869460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657869460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,524 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:49,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:49,528 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:49,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:49,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:49,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,538 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:49,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:49,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:49,552 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:49,552 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,552 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:49,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:49,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:49,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:49,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:49,560 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:49,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:49,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:49,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:49,591 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/48e202533de845078553984241c445e2 2023-04-28 04:56:49,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:49,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2acd0dc0[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:49,603 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:49,603 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:49,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:49,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:49,622 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,622 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:49,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:49,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,633 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:49,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:49,636 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:49,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:49,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:49,636 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:49,636 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:49,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3 2023-04-28 04:56:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,645 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,649 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:49,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:49,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,664 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:49,664 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/12d7e64b66ce481da16753ec24dffb8a 2023-04-28 04:56:49,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/48e202533de845078553984241c445e2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/48e202533de845078553984241c445e2 2023-04-28 04:56:49,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/48e202533de845078553984241c445e2, entries=200, sequenceid=150, filesize=14.1 K 2023-04-28 04:56:49,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3 2023-04-28 04:56:49,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3, entries=150, sequenceid=150, filesize=11.8 K 2023-04-28 04:56:49,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/12d7e64b66ce481da16753ec24dffb8a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/12d7e64b66ce481da16753ec24dffb8a 2023-04-28 04:56:49,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/12d7e64b66ce481da16753ec24dffb8a, entries=150, sequenceid=150, filesize=11.8 K 2023-04-28 04:56:49,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=33.54 KB/34350 for bf0b022e81bb29fa4ec0f9acedc0e278 in 568ms, sequenceid=150, compaction requested=true 2023-04-28 04:56:49,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:49,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:49,708 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:49,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:49,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 04:56:49,709 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52527 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:49,710 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:56:49,710 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,710 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3a6d9dd852fa413aaf97eb3f23bea0b9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/d5b497cd3d9c455db808e9308669936c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/55ee5f3d0d3948928465c55df69a2495, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/48e202533de845078553984241c445e2] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=51.3 K 2023-04-28 04:56:49,710 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3a6d9dd852fa413aaf97eb3f23bea0b9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682657806158 2023-04-28 04:56:49,710 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:49,711 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d5b497cd3d9c455db808e9308669936c, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682657806210 2023-04-28 04:56:49,711 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 55ee5f3d0d3948928465c55df69a2495, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657806825 2023-04-28 04:56:49,712 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 48e202533de845078553984241c445e2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657808006 2023-04-28 04:56:49,713 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:49,713 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:56:49,713 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,713 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c0fd289d1d184e079137a6129373dcd9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/03152deba7e74dc18ee2aafcfdd2fa41, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f8ca860f164070aed3e1ea42834907, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=46.7 K 2023-04-28 04:56:49,714 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c0fd289d1d184e079137a6129373dcd9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682657806158 2023-04-28 04:56:49,714 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 03152deba7e74dc18ee2aafcfdd2fa41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682657806210 2023-04-28 04:56:49,715 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 41f8ca860f164070aed3e1ea42834907, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657806825 2023-04-28 04:56:49,715 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5d1a5f5432b44a37b8b05d8e6cfb1cb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657808006 2023-04-28 04:56:49,722 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:49,744 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#195 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:49,765 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:49,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:49,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:49,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:49,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:49,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:49,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:56:49,769 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:49,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:49,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:49,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:49,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,780 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:49,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:49,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:49,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:49,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:49,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:49,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:49,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:49,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682657809778, max timestamp=1682657809789Num uniques 859; 2023-04-28 04:56:49,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:49,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:49,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f37505d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:49,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657869796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,801 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:49,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:49,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:49,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:49,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657869796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657869797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657869797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:49,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:49,813 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:49,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:49,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:49,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:49,813 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:49,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,822 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,832 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:49,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:49,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,833 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:49,833 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657869900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657869902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657869903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:49,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657869903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,933 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:49,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:49,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:49,936 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:49,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:49,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:49,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:49,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:49,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:49,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:49,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:49,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:49,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:49,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:49,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:49,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:49,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:49,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:49,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:49,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,959 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:49,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:49,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69e3cdc0[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:49,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:49,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:49,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:49,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:49,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:49,980 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:49,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:49,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:49,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:49,984 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:49,985 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:49,985 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:49,985 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:49,985 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:49,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:49,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:49,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:49,991 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,991 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:50,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:50,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,002 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,002 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:50,102 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:50,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:50,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:50,104 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:50,104 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,104 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:50,104 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:50,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657870105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657870105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657870106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657870106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,117 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:50,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:50,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:50,118 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,118 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:50,118 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:50,118 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:50,118 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:50,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:50,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417152b8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:50,129 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,165 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/3f30a903cba2448cac29bbf4ddaae50f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3f30a903cba2448cac29bbf4ddaae50f 2023-04-28 04:56:50,168 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/6426b121ec254b349275181bf2afd4c0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6426b121ec254b349275181bf2afd4c0 2023-04-28 04:56:50,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:50,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:50,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:50,170 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into 3f30a903cba2448cac29bbf4ddaae50f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:50,170 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:50,170 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=12, startTime=1682657809708; duration=0sec 2023-04-28 04:56:50,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:50,170 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:50,170 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:50,172 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:50,172 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:56:50,173 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,173 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1af301275a44886ad0a06b32d9e3d48, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/5dc4e995f26e4e71a3c092da99d35384, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/1ea0fa0d0f7444cd94e480784766eb62, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/12d7e64b66ce481da16753ec24dffb8a] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=46.7 K 2023-04-28 04:56:50,173 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b1af301275a44886ad0a06b32d9e3d48, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682657806158 2023-04-28 04:56:50,175 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into 6426b121ec254b349275181bf2afd4c0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:50,175 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:50,175 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=12, startTime=1682657809708; duration=0sec 2023-04-28 04:56:50,175 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:50,180 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5dc4e995f26e4e71a3c092da99d35384, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682657806210 2023-04-28 04:56:50,180 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 1ea0fa0d0f7444cd94e480784766eb62, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657806825 2023-04-28 04:56:50,180 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 12d7e64b66ce481da16753ec24dffb8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657808006 2023-04-28 04:56:50,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/4947e23b93984e0b90da9c4eb087b560 2023-04-28 04:56:50,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:50,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:50,191 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:50,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:50,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:50,194 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:50,195 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:50,195 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:50,195 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:50,195 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:50,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,203 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/2f79bd25a48c46a381dda654fd8ce026 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2f79bd25a48c46a381dda654fd8ce026 2023-04-28 04:56:50,203 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,211 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into 2f79bd25a48c46a381dda654fd8ce026(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:50,211 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:50,211 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=12, startTime=1682657809708; duration=0sec 2023-04-28 04:56:50,211 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:50,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:50,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:50,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,212 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:50,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,212 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:50,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,312 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:50,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:50,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:50,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:50,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:50,315 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:50,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:50,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:50,316 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,316 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,327 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:50,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:50,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:50,332 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:50,332 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,332 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:50,334 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:50,334 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,334 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:50,334 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:50,334 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,334 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:50,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:50,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:50,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:50,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:50,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:50,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fa80fe[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:50,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:50,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:50,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:50,359 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,359 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:50,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:50,359 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:50,359 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:50,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657870363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,367 DEBUG [Thread-1408] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:50,369 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:50,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:50,374 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:50,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:50,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:50,374 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:50,374 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:50,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,382 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,385 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:50,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:50,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,391 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:50,391 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:50,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:50,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,391 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,392 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,392 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657870408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657870408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657870410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657870410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,491 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:50,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:50,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:50,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:50,504 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:50,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:50,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:50,505 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,505 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,517 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:50,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:50,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:50,520 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:50,520 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,520 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:50,530 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:50,531 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:50,531 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:50,531 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:50,531 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:50,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:50,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:50,531 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:50,538 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,538 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@390f45b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:50,540 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:50,540 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,548 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:50,548 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:50,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:50,549 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,549 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:50,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:50,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:50,560 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:50,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:50,565 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:50,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:50,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:50,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:50,565 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:50,569 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,569 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,569 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,570 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,572 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:50,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,580 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:50,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:50,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:50,581 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:50,581 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:50,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:50,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:50,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/1ee95c570fbe471c9b2956835565420c 2023-04-28 04:56:50,681 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:50,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:50,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:50,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:50,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:50,688 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:50,688 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,688 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:50,688 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:50,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:50,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:50,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:50,718 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:50,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:50,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:50,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:50,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:50,732 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:50,736 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,736 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:50,736 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:50,736 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:50,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:50,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:50,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,806 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:50,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:50,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5670f9cc[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:50,809 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:50,809 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657870912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657870916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657870916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657870917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:50,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:50,949 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:50,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:50,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:50,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:51,023 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:51,023 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,023 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:51,023 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:51,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:51,026 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:51,026 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:51,026 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:51,026 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:51,026 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:51,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:51,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:51,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:51,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:51,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,053 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:51,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/79659cc293604eb7b9bafc40e78307a2 2023-04-28 04:56:51,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:51,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,054 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:51,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:51,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:51,054 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:51,054 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:51,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/4947e23b93984e0b90da9c4eb087b560 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4947e23b93984e0b90da9c4eb087b560 2023-04-28 04:56:51,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4947e23b93984e0b90da9c4eb087b560, entries=150, sequenceid=160, filesize=11.8 K 2023-04-28 04:56:51,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/1ee95c570fbe471c9b2956835565420c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1ee95c570fbe471c9b2956835565420c 2023-04-28 04:56:51,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1ee95c570fbe471c9b2956835565420c, entries=150, sequenceid=160, filesize=11.8 K 2023-04-28 04:56:51,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/79659cc293604eb7b9bafc40e78307a2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79659cc293604eb7b9bafc40e78307a2 2023-04-28 04:56:51,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79659cc293604eb7b9bafc40e78307a2, entries=150, sequenceid=160, filesize=11.8 K 2023-04-28 04:56:51,072 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1304ms, sequenceid=160, compaction requested=false 2023-04-28 04:56:51,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:51,154 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:51,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:51,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:51,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:51,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:51,157 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:51,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:51,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:51,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:51,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:51,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:51,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:51,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:51,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:51,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:51,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:51,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:51,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:51,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:51,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:51,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:51,171 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:51,171 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=131.94 KB heapSize=386.72 KB 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:51,171 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:51,177 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/9d799c3413da48ff88a5d670fa32a8ee 2023-04-28 04:56:51,587 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/12f3ecc0bb394ef59c4454bdf254e0e1 2023-04-28 04:56:51,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:51,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:51,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:51,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:51,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657871931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657871931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:51,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657871932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657871933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:51,998 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/26aba9573eff4e2f8b9f4787e40efdc2 2023-04-28 04:56:52,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/9d799c3413da48ff88a5d670fa32a8ee as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/9d799c3413da48ff88a5d670fa32a8ee 2023-04-28 04:56:52,007 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/9d799c3413da48ff88a5d670fa32a8ee, entries=150, sequenceid=190, filesize=11.8 K 2023-04-28 04:56:52,008 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/12f3ecc0bb394ef59c4454bdf254e0e1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/12f3ecc0bb394ef59c4454bdf254e0e1 2023-04-28 04:56:52,012 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/12f3ecc0bb394ef59c4454bdf254e0e1, entries=150, sequenceid=190, filesize=11.8 K 2023-04-28 04:56:52,015 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/26aba9573eff4e2f8b9f4787e40efdc2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/26aba9573eff4e2f8b9f4787e40efdc2 2023-04-28 04:56:52,029 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/26aba9573eff4e2f8b9f4787e40efdc2, entries=150, sequenceid=190, filesize=11.8 K 2023-04-28 04:56:52,030 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~386.67 KB/395952, currentSize=46.96 KB/48090 for bf0b022e81bb29fa4ec0f9acedc0e278 in 859ms, sequenceid=190, compaction requested=true 2023-04-28 04:56:52,031 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:52,031 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,031 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:52,031 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:52,031 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:52,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:52,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 04:56:52,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:52,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:52,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:52,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:52,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:52,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:52,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@218ab30f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:52,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:52,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657872067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657872069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657872070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657872071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:52,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:52,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:52,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:52,138 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,138 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:52,138 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,138 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:52,140 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:52,140 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:52,140 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:52,140 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:52,140 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:52,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,149 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,151 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:52,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,160 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:52,159 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,160 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,160 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,160 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657872172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657872174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657872174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657872174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,260 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:52,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:52,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:52,262 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:52,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:52,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:52,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:52,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:52,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:52,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:52,288 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:52,288 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:52,290 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,290 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:52,291 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:52,291 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:52,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:52,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:52,291 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:52,296 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,296 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,299 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,299 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:52,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:52,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa5c114[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:52,300 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,306 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:52,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:52,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:52,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:52,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,326 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,326 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:52,329 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:52,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:52,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:52,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:52,330 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:52,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,338 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,338 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,338 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,341 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:52,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,349 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:52,349 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:52,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:52,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657872376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657872377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657872378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657872378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,449 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:52,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:52,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:52,460 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:52,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:52,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:52,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:52,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:52,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:52,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:52,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:52,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:52,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:52,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:52,476 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:52,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/7590837770ee4fc4ad064ec90100f055 2023-04-28 04:56:52,485 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,486 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,486 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:52,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e13c237[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:52,488 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:52,488 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:52,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:52,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:52,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:52,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:52,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:52,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:52,511 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:52,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:52,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:52,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:52,511 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:52,527 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,527 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,527 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,528 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,570 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:52,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,570 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:52,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:52,570 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,570 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,670 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:52,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:52,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:52,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:52,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:52,673 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:52,673 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,673 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:52,673 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:52,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657872680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657872680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657872681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:52,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657872682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,695 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:52,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:52,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:52,697 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,697 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:52,697 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:52,697 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:52,697 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:52,706 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:52,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7ce8a0[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:52,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:52,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,717 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,717 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:52,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:52,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:52,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:52,727 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:52,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,727 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,727 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:52,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:52,729 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:52,730 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:52,730 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:52,730 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:52,730 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:52,737 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,738 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,738 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,738 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:52,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,749 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:52,749 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,749 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,749 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,749 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,749 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:52,749 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:52,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,849 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:52,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:52,852 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:52,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:52,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:52,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,864 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,864 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,864 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,864 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:52,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:52,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:52,865 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:52,865 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:52,865 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:52,865 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:52,865 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:52,865 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,865 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:52,866 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:52,866 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:52,866 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:52,866 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:52,866 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,866 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:52,866 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:52,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,874 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1a268b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:52,876 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:52,876 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:52,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:52,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:52,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:52,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:52,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:52,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:52,897 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:52,897 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:52,897 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:52,897 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:52,897 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:52,906 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,906 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:52,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/a01807738df74d4f9bc64e8db77e44e6 2023-04-28 04:56:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,906 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:52,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:52,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:52,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:52,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:52,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:52,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:52,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:52,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:52,917 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:52,917 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:52,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:52,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:53,017 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:53,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:53,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:53,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:53,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:53,024 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:53,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:53,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:53,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:53,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,032 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:53,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:53,033 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,033 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:53,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:53,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:53,034 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,034 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:53,034 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:53,034 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:53,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:53,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:53,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:53,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:53,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f73f39d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:53,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:53,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:53,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:53,054 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:53,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:53,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:53,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:53,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:53,066 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:53,066 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:53,066 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:53,066 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:53,066 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:53,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:53,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:53,075 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,112 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,112 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:53,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:53,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:53,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:53,112 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:53,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:53,112 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:53,112 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,113 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:53,113 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:53,113 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,113 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:53,113 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,113 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,113 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:53,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657873184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657873186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657873186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:53,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657873190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,213 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:53,216 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:53,216 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:53,216 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:53,216 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:53,217 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,217 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,232 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,232 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:53,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:53,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:53,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:53,234 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,234 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:53,234 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:53,234 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:53,234 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:53,243 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:53,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:53,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:53,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:53,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f818a3e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:53,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,253 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:53,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:53,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:53,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:53,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:53,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:53,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:53,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:53,266 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:53,266 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:53,266 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:53,266 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:53,266 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:53,285 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,285 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,285 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:53,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:53,286 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,287 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:53,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:53,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:53,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:53,296 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:53,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:53,296 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:53,296 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:53,296 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:53,296 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:53,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/b5c861ecdf4847049ca9da96f4a5658c 2023-04-28 04:56:53,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/7590837770ee4fc4ad064ec90100f055 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7590837770ee4fc4ad064ec90100f055 2023-04-28 04:56:53,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7590837770ee4fc4ad064ec90100f055, entries=150, sequenceid=203, filesize=11.8 K 2023-04-28 04:56:53,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/a01807738df74d4f9bc64e8db77e44e6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/a01807738df74d4f9bc64e8db77e44e6 2023-04-28 04:56:53,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/a01807738df74d4f9bc64e8db77e44e6, entries=150, sequenceid=203, filesize=11.8 K 2023-04-28 04:56:53,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/b5c861ecdf4847049ca9da96f4a5658c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b5c861ecdf4847049ca9da96f4a5658c 2023-04-28 04:56:53,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b5c861ecdf4847049ca9da96f4a5658c, entries=150, sequenceid=203, filesize=11.8 K 2023-04-28 04:56:53,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1291ms, sequenceid=203, compaction requested=true 2023-04-28 04:56:53,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:53,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:53,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:53,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:53,331 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:53,331 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:53,332 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:53,332 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:53,332 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:56:53,333 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:56:53,333 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,333 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,333 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3f30a903cba2448cac29bbf4ddaae50f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4947e23b93984e0b90da9c4eb087b560, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/9d799c3413da48ff88a5d670fa32a8ee, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7590837770ee4fc4ad064ec90100f055] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=47.3 K 2023-04-28 04:56:53,333 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6426b121ec254b349275181bf2afd4c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1ee95c570fbe471c9b2956835565420c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/12f3ecc0bb394ef59c4454bdf254e0e1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/a01807738df74d4f9bc64e8db77e44e6] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=47.3 K 2023-04-28 04:56:53,333 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3f30a903cba2448cac29bbf4ddaae50f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657808006 2023-04-28 04:56:53,333 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 6426b121ec254b349275181bf2afd4c0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657808006 2023-04-28 04:56:53,333 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4947e23b93984e0b90da9c4eb087b560, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682657809147 2023-04-28 04:56:53,333 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1ee95c570fbe471c9b2956835565420c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682657809147 2023-04-28 04:56:53,334 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9d799c3413da48ff88a5d670fa32a8ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682657809790 2023-04-28 04:56:53,334 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 12f3ecc0bb394ef59c4454bdf254e0e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682657809790 2023-04-28 04:56:53,334 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7590837770ee4fc4ad064ec90100f055, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682657811930 2023-04-28 04:56:53,334 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a01807738df74d4f9bc64e8db77e44e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682657811930 2023-04-28 04:56:53,343 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#206 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:53,346 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#207 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:53,381 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/65b1ad830b3a4387a32993457b123138 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/65b1ad830b3a4387a32993457b123138 2023-04-28 04:56:53,387 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into 65b1ad830b3a4387a32993457b123138(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:53,387 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:53,387 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=12, startTime=1682657813331; duration=0sec 2023-04-28 04:56:53,388 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:53,388 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:56:53,390 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:56:53,390 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:56:53,390 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,390 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2f79bd25a48c46a381dda654fd8ce026, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79659cc293604eb7b9bafc40e78307a2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/26aba9573eff4e2f8b9f4787e40efdc2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b5c861ecdf4847049ca9da96f4a5658c] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=47.3 K 2023-04-28 04:56:53,390 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 2f79bd25a48c46a381dda654fd8ce026, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682657808006 2023-04-28 04:56:53,391 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 79659cc293604eb7b9bafc40e78307a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682657809147 2023-04-28 04:56:53,391 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 26aba9573eff4e2f8b9f4787e40efdc2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682657809790 2023-04-28 04:56:53,391 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b5c861ecdf4847049ca9da96f4a5658c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682657811930 2023-04-28 04:56:53,396 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:53,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:53,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:53,400 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:53,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:53,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:53,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:53,401 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,401 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,409 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:53,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:53,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:53,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:53,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:53,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:53,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:53,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:53,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:53,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:53,430 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:53,460 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:53,460 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:53,460 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:53,460 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:53,460 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:56:53,461 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:53,461 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:53,461 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:53,461 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:53,461 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:53,461 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:53,466 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/2b870a680d4c47988617010187443acd as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2b870a680d4c47988617010187443acd 2023-04-28 04:56:53,474 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into 2b870a680d4c47988617010187443acd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:53,474 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:53,474 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=12, startTime=1682657813331; duration=0sec 2023-04-28 04:56:53,474 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:53,767 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/8764e0324cc94427badf34f1a505e4b1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/8764e0324cc94427badf34f1a505e4b1 2023-04-28 04:56:53,771 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into 8764e0324cc94427badf34f1a505e4b1(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:53,771 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:53,771 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=12, startTime=1682657813331; duration=0sec 2023-04-28 04:56:53,772 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:53,882 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/121bd058a3bb41da9f96956050b01681 2023-04-28 04:56:54,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:54,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:54,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657874201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657874215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657874215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657874215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,291 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/ff8f88a9bf004336abc51fb9d506fd4f 2023-04-28 04:56:54,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657874317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657874317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657874318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657874318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58532 deadline: 1682657874368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,370 DEBUG [Thread-1408] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:56:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657874522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657874522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657874522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657874522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,701 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/c99a94a55ecf463895ec147ae8aa20b8 2023-04-28 04:56:54,705 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/121bd058a3bb41da9f96956050b01681 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/121bd058a3bb41da9f96956050b01681 2023-04-28 04:56:54,708 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/121bd058a3bb41da9f96956050b01681, entries=150, sequenceid=224, filesize=11.8 K 2023-04-28 04:56:54,709 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/ff8f88a9bf004336abc51fb9d506fd4f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/ff8f88a9bf004336abc51fb9d506fd4f 2023-04-28 04:56:54,717 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/ff8f88a9bf004336abc51fb9d506fd4f, entries=150, sequenceid=224, filesize=11.8 K 2023-04-28 04:56:54,717 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/c99a94a55ecf463895ec147ae8aa20b8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/c99a94a55ecf463895ec147ae8aa20b8 2023-04-28 04:56:54,722 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/c99a94a55ecf463895ec147ae8aa20b8, entries=150, sequenceid=224, filesize=11.8 K 2023-04-28 04:56:54,723 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1262ms, sequenceid=224, compaction requested=false 2023-04-28 04:56:54,723 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:54,723 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:54,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:54,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:54,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:54,723 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:54,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:54,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,791 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:54,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:54,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27e04486[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:54,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:54,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,801 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:54,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:54,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:54,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:54,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:54,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:54,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:54,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:54,813 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:54,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:54,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:54,813 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:54,813 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:54,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:54,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:54,822 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:54,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:56:54,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:54,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:54,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:54,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:54,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:54,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:54,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:54,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:54,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:54,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:54,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:54,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:54,832 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:54,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:54,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:54,833 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:54,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,833 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:54,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,833 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657874857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657874857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657874858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657874859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,933 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:54,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:54,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:54,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:54,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:54,936 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:54,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:54,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:54,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:54,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:54,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:54,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:54,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:54,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:54,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:54,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:54,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:54,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:54,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:54,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:54,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:54,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:54,949 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:54,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:54,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:54,950 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:54,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:54,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:54,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:54,950 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:54,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:54,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,959 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:54,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:54,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:54,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd4008f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:54,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657874959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657874961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657874961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657874962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:54,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:54,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:54,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:54,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:54,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:54,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:54,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:54,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:54,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:54,983 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:54,983 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:54,983 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:54,983 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:54,983 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:54,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:54,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:54,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:54,994 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,995 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:55,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:55,002 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:55,002 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:55,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,002 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,102 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:55,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:55,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:55,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:55,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:55,111 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:55,111 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,111 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:55,111 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:55,111 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,111 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,147 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,147 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,147 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:55,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:55,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:55,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:55,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:55,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:55,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:55,148 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,149 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:55,149 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:55,149 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,149 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:55,149 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:55,149 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,149 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:55,149 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:55,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657875163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657875164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657875165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58494 deadline: 1682657875165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:55,171 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:55,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dabe3b3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:55,171 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:55,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:55,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:55,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:55,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:55,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:55,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:55,192 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:55,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:55,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:55,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:55,192 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:55,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,201 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:55,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:55,211 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:55,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,211 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:55,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/05ad2f973da941e3835c84f11b086098 2023-04-28 04:56:55,312 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:55,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:55,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:55,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:55,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:55,314 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:55,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:55,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:55,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,327 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:55,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:55,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:55,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:55,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:55,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:55,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:55,329 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,329 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:55,329 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:55,329 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:55,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:55,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:55,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:55,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:55,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e884c42[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:55,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:55,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,348 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:55,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:55,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:55,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:55,358 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:55,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,358 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,359 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:55,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:55,360 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:55,360 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:55,360 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:55,360 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:55,360 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:55,369 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,369 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,369 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,369 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,371 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,380 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:55,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:55,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:55,380 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:55,380 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:55,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,380 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657875466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657875467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657875467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657875468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,480 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:55,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:55,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:55,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:55,483 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:55,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:55,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:55,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:55,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:55,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:55,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:55,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:55,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:55,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:55,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:55,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be9e761[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:55,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,516 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:55,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:55,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:55,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:55,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:55,527 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:55,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:55,529 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:55,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:55,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:55,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:55,529 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:55,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,539 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:55,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,548 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,549 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:55,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,649 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:55,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:55,651 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:55,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:55,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:55,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/6c381c65f60046758ccd0119d756738d 2023-04-28 04:56:55,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:55,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:55,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:55,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:55,666 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,666 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:55,666 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:55,666 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:55,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:55,706 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:55,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:55,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662440eb[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:55,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,727 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:55,727 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,727 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:55,727 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:55,727 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:55,737 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:55,737 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,737 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,738 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:55,739 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:55,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:55,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:55,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:55,739 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:55,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,748 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,749 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:55,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:55,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,759 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:55,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,759 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:55,759 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,759 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,759 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,859 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:55,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:55,862 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:55,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:55,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:55,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,874 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,874 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,875 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,875 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:55,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:55,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:55,876 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,876 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:55,876 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:55,876 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:55,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:55,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:55,886 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14e081e7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:55,886 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,896 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:55,896 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:55,896 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:55,896 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,896 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:55,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:55,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:55,906 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:55,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:55,908 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:55,908 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:55,908 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:55,908 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:55,908 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:55,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:55,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:55,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:55,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:55,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:55,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:55,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:55,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:55,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:55,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:55,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,927 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:55,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,928 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:55,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:55,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:55,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:55,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58496 deadline: 1682657875970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657875971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58506 deadline: 1682657875971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:55,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:58498 deadline: 1682657875971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:56,028 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:56,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:56,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:56,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:56,030 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:56,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:56,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:56,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:56,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:56,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:56,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/b1bfbec945a14d4f9fe0588d59151a78 2023-04-28 04:56:56,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/05ad2f973da941e3835c84f11b086098 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/05ad2f973da941e3835c84f11b086098 2023-04-28 04:56:56,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/05ad2f973da941e3835c84f11b086098, entries=150, sequenceid=239, filesize=11.8 K 2023-04-28 04:56:56,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/6c381c65f60046758ccd0119d756738d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6c381c65f60046758ccd0119d756738d 2023-04-28 04:56:56,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6c381c65f60046758ccd0119d756738d, entries=150, sequenceid=239, filesize=11.8 K 2023-04-28 04:56:56,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/b1bfbec945a14d4f9fe0588d59151a78 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1bfbec945a14d4f9fe0588d59151a78 2023-04-28 04:56:56,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1bfbec945a14d4f9fe0588d59151a78, entries=150, sequenceid=239, filesize=11.8 K 2023-04-28 04:56:56,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1262ms, sequenceid=239, compaction requested=true 2023-04-28 04:56:56,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:56,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:56,087 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:56,087 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:56,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:56,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:56,088 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:56,088 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:56,088 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:56:56,088 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:56:56,088 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:56,088 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:56,088 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/65b1ad830b3a4387a32993457b123138, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/121bd058a3bb41da9f96956050b01681, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/05ad2f973da941e3835c84f11b086098] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=35.6 K 2023-04-28 04:56:56,088 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/8764e0324cc94427badf34f1a505e4b1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/ff8f88a9bf004336abc51fb9d506fd4f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6c381c65f60046758ccd0119d756738d] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=35.6 K 2023-04-28 04:56:56,088 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 65b1ad830b3a4387a32993457b123138, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682657811930 2023-04-28 04:56:56,089 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8764e0324cc94427badf34f1a505e4b1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682657811930 2023-04-28 04:56:56,089 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 121bd058a3bb41da9f96956050b01681, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682657812066 2023-04-28 04:56:56,089 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ff8f88a9bf004336abc51fb9d506fd4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682657812066 2023-04-28 04:56:56,089 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 05ad2f973da941e3835c84f11b086098, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682657814192 2023-04-28 04:56:56,089 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 6c381c65f60046758ccd0119d756738d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682657814192 2023-04-28 04:56:56,098 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:56,099 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:56,108 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/c9b622312af34340b288f3e9c4121b74 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c9b622312af34340b288f3e9c4121b74 2023-04-28 04:56:56,112 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into c9b622312af34340b288f3e9c4121b74(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:56,113 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:56,113 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=13, startTime=1682657816087; duration=0sec 2023-04-28 04:56:56,113 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:56,113 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:56,113 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:56,113 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:56,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:56,114 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:56,114 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:56,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:56,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:56,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:56,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:56,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:56,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:56,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:56,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:56,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:56,115 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:56,115 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:56,115 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 04:56:56,116 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:56,116 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:56,116 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:56,116 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:56,116 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:56,116 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:56,124 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:56,125 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:56:56,125 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:56,125 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2b870a680d4c47988617010187443acd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/c99a94a55ecf463895ec147ae8aa20b8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1bfbec945a14d4f9fe0588d59151a78] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=35.6 K 2023-04-28 04:56:56,125 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 2b870a680d4c47988617010187443acd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682657811930 2023-04-28 04:56:56,126 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c99a94a55ecf463895ec147ae8aa20b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682657812066 2023-04-28 04:56:56,126 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b1bfbec945a14d4f9fe0588d59151a78, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682657814192 2023-04-28 04:56:56,145 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:56,157 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/3f616a1b22334edf963dbdee08bbf191 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3f616a1b22334edf963dbdee08bbf191 2023-04-28 04:56:56,161 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into 3f616a1b22334edf963dbdee08bbf191(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:56,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:56,161 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=13, startTime=1682657816087; duration=0sec 2023-04-28 04:56:56,161 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:56,541 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/6d027f5073794d6e9e1559b4f57e0e7c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6d027f5073794d6e9e1559b4f57e0e7c 2023-04-28 04:56:56,546 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into 6d027f5073794d6e9e1559b4f57e0e7c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:56,547 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:56,547 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=13, startTime=1682657816087; duration=0sec 2023-04-28 04:56:56,547 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:56,550 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/91a235d7071e45d99762fbf3288a8e19 2023-04-28 04:56:56,959 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/b6a25ccadbec4f0eb66c26aec557ac4a 2023-04-28 04:56:56,969 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/f1ec80ceb4da47edb0a37c8a16068da6 2023-04-28 04:56:56,973 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/91a235d7071e45d99762fbf3288a8e19 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/91a235d7071e45d99762fbf3288a8e19 2023-04-28 04:56:56,976 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/91a235d7071e45d99762fbf3288a8e19, entries=150, sequenceid=259, filesize=11.8 K 2023-04-28 04:56:56,976 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/b6a25ccadbec4f0eb66c26aec557ac4a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/b6a25ccadbec4f0eb66c26aec557ac4a 2023-04-28 04:56:56,980 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/b6a25ccadbec4f0eb66c26aec557ac4a, entries=150, sequenceid=259, filesize=11.8 K 2023-04-28 04:56:56,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:56,981 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:56,981 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/f1ec80ceb4da47edb0a37c8a16068da6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f1ec80ceb4da47edb0a37c8a16068da6 2023-04-28 04:56:56,984 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f1ec80ceb4da47edb0a37c8a16068da6, entries=150, sequenceid=259, filesize=11.8 K 2023-04-28 04:56:56,985 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=26.84 KB/27480 for bf0b022e81bb29fa4ec0f9acedc0e278 in 870ms, sequenceid=259, compaction requested=false 2023-04-28 04:56:56,985 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:56,985 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:56,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:56,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:56,986 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:56,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:56,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:56:56,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:56,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:56,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:56,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:56,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:56,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:56,988 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:56,988 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:57,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657817002, max timestamp=1682657817007Num uniques 859; 2023-04-28 04:56:57,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:57,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657817002, max timestamp=1682657817007Num uniques 859; 2023-04-28 04:56:57,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:57,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657817002, max timestamp=1682657817007Num uniques 859; 2023-04-28 04:56:57,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:56:57,014 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,014 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,014 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,014 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:57,015 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8963565[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:57,015 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,032 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:57,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:57,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:57,033 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,033 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:57,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657877040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657877041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657877041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657877041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:57,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:57,045 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:57,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:57,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:57,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:57,045 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:57,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,064 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:57,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:57,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:57,064 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,064 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:57,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657877144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657877144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657877144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657877144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,164 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:57,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:57,167 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:57,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:57,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:57,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:57,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:57,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:57,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:57,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:57,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:57,181 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,181 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:57,181 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:57,181 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:57,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:57,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:57,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7019f9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:57,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:57,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:57,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:57,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:57,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:57,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:57,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:57,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:57,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:57,246 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:57,246 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:57,246 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:57,246 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:57,246 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:57,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:57,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,265 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:57,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:57,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:57,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:57,275 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:57,275 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:57,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:57,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:57,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657877346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657877347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657877347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657877347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,375 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:57,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:57,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:57,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:57,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:57,378 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:57,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:57,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:57,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:57,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:57,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:57,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:57,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:57,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:57,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:57,391 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:57,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:57,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:57,392 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:57,392 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:57,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/eed7161755184ad88da593eff32f3e34 2023-04-28 04:56:57,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/530078fd48414282bff3fa976560e857 2023-04-28 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78173584[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:57,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:57,403 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/09f925c65a3048549ad7adb55993f98a 2023-04-28 04:56:57,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:57,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:57,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:57,411 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,411 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:57,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:57,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/eed7161755184ad88da593eff32f3e34 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/eed7161755184ad88da593eff32f3e34 2023-04-28 04:56:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:57,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:57,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:57,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:57,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:57,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:57,425 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:57,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/eed7161755184ad88da593eff32f3e34, entries=300, sequenceid=271, filesize=19.0 K 2023-04-28 04:56:57,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/530078fd48414282bff3fa976560e857 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/530078fd48414282bff3fa976560e857 2023-04-28 04:56:57,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/530078fd48414282bff3fa976560e857, entries=150, sequenceid=271, filesize=11.9 K 2023-04-28 04:56:57,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/09f925c65a3048549ad7adb55993f98a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/09f925c65a3048549ad7adb55993f98a 2023-04-28 04:56:57,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:57,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:57,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:57,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/09f925c65a3048549ad7adb55993f98a, entries=150, sequenceid=271, filesize=11.9 K 2023-04-28 04:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for bf0b022e81bb29fa4ec0f9acedc0e278 in 450ms, sequenceid=271, compaction requested=true 2023-04-28 04:56:57,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:57,436 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:57,436 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:57,442 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44106 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:57,443 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:56:57,443 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,443 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6d027f5073794d6e9e1559b4f57e0e7c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/91a235d7071e45d99762fbf3288a8e19, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/eed7161755184ad88da593eff32f3e34] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=43.1 K 2023-04-28 04:56:57,443 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:57,443 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:56:57,443 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,443 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c9b622312af34340b288f3e9c4121b74, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/b6a25ccadbec4f0eb66c26aec557ac4a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/530078fd48414282bff3fa976560e857] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=35.9 K 2023-04-28 04:56:57,444 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 6d027f5073794d6e9e1559b4f57e0e7c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682657814192 2023-04-28 04:56:57,444 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c9b622312af34340b288f3e9c4121b74, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682657814192 2023-04-28 04:56:57,444 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 91a235d7071e45d99762fbf3288a8e19, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682657814854 2023-04-28 04:56:57,445 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting eed7161755184ad88da593eff32f3e34, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657816979 2023-04-28 04:56:57,445 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b6a25ccadbec4f0eb66c26aec557ac4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682657814854 2023-04-28 04:56:57,445 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 530078fd48414282bff3fa976560e857, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657816984 2023-04-28 04:56:57,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:57,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:57,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:57,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:57,454 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:57,454 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:57,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:57,455 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:57,453 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:57,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:57,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,456 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,456 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,456 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,456 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,456 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:57,456 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:57,554 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:57,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:57,556 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:57,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:57,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:57,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:57,557 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,557 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,569 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:57,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:57,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:57,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:57,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:57,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:57,570 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:57,570 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:57,570 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:57,571 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:57,571 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:57,571 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,571 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:57,571 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-28 04:56:57,571 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:57,572 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:57,572 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:57,572 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:57,572 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:57,572 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:56:57,582 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/37c96da1aa5242dd8feed450ab1c8c15 2023-04-28 04:56:57,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:57,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:57,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657877655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657877656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657877657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657877659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657877762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657877762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657877762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657877763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,887 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/968d83a1686f42c1ad90cf77027cb2d1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/968d83a1686f42c1ad90cf77027cb2d1 2023-04-28 04:56:57,893 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into 968d83a1686f42c1ad90cf77027cb2d1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:57,893 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:57,893 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=13, startTime=1682657817436; duration=0sec 2023-04-28 04:56:57,894 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:56:57,894 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:56:57,894 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/5d257fc4a9994c24a4690b02e109f565 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/5d257fc4a9994c24a4690b02e109f565 2023-04-28 04:56:57,895 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:56:57,895 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:56:57,895 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:57,895 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3f616a1b22334edf963dbdee08bbf191, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f1ec80ceb4da47edb0a37c8a16068da6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/09f925c65a3048549ad7adb55993f98a] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=35.9 K 2023-04-28 04:56:57,897 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3f616a1b22334edf963dbdee08bbf191, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1682657814192 2023-04-28 04:56:57,900 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f1ec80ceb4da47edb0a37c8a16068da6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682657814854 2023-04-28 04:56:57,901 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 09f925c65a3048549ad7adb55993f98a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657816984 2023-04-28 04:56:57,903 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into 5d257fc4a9994c24a4690b02e109f565(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:57,903 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:57,903 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=13, startTime=1682657817436; duration=0sec 2023-04-28 04:56:57,904 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:57,909 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:56:57,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657877965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657877965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657877966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:57,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657877966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:57,992 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/6bd83c40d4504c2797ddab7befd96401 2023-04-28 04:56:58,030 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/3fd8cfd8646b4e9ab4f7533b87b317fa 2023-04-28 04:56:58,045 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/37c96da1aa5242dd8feed450ab1c8c15 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/37c96da1aa5242dd8feed450ab1c8c15 2023-04-28 04:56:58,049 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/37c96da1aa5242dd8feed450ab1c8c15, entries=150, sequenceid=307, filesize=11.9 K 2023-04-28 04:56:58,050 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/6bd83c40d4504c2797ddab7befd96401 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6bd83c40d4504c2797ddab7befd96401 2023-04-28 04:56:58,054 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6bd83c40d4504c2797ddab7befd96401, entries=150, sequenceid=307, filesize=11.9 K 2023-04-28 04:56:58,054 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/3fd8cfd8646b4e9ab4f7533b87b317fa as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3fd8cfd8646b4e9ab4f7533b87b317fa 2023-04-28 04:56:58,060 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3fd8cfd8646b4e9ab4f7533b87b317fa, entries=150, sequenceid=307, filesize=11.9 K 2023-04-28 04:56:58,061 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for bf0b022e81bb29fa4ec0f9acedc0e278 in 490ms, sequenceid=307, compaction requested=false 2023-04-28 04:56:58,061 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:58,061 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:58,061 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:58,061 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:58,061 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,061 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:58,061 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:58,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:58,106 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,106 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:58,106 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:58,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:58,110 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:58,110 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e87cd8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:58,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,117 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:58,117 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:58,117 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:58,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:58,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:58,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:58,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:58,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:58,130 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:58,130 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:58,130 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:58,130 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:58,130 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:58,137 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,137 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,137 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:58,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:58,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:58,138 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,139 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:58,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:58,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:58,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:58,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:58,148 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:58,148 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:58,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:58,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:58,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:58,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:58,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:58,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,249 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:58,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:58,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:58,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:58,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:58,251 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:58,251 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:58,251 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:58,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:58,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:58,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:58,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:58,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:58,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:58,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:58,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:58,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:58,265 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:58,265 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:58,265 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:58,267 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:58,267 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:58,267 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:58,267 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:58,267 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 04:56:58,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:58,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:58,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:58,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:58,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:58,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:56:58,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:58,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:58,277 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/7ac4e69a5b9448cfad5365865b5a1ea2 2023-04-28 04:56:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657818287, max timestamp=1682657818292Num uniques 859; 2023-04-28 04:56:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657818287, max timestamp=1682657818292Num uniques 859; 2023-04-28 04:56:58,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:58,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682657818287, max timestamp=1682657818292Num uniques 859; 2023-04-28 04:56:58,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:56:58,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657878309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657878310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657878311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657878312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,330 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/0ae7ce84895c4110a4eeda8b7855e47d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/0ae7ce84895c4110a4eeda8b7855e47d 2023-04-28 04:56:58,333 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into 0ae7ce84895c4110a4eeda8b7855e47d(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:56:58,333 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:58,334 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=13, startTime=1682657817436; duration=0sec 2023-04-28 04:56:58,334 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:56:58,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657878413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657878414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657878415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657878415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657878616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657878618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657878619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657878619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,698 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5524ca50728e43bda4cf01d6589461d3 2023-04-28 04:56:58,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657878920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657878920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657878923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:58,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657878923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,109 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/6affbab971f44450aedcbd99a278de91 2023-04-28 04:56:59,113 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/7ac4e69a5b9448cfad5365865b5a1ea2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7ac4e69a5b9448cfad5365865b5a1ea2 2023-04-28 04:56:59,116 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7ac4e69a5b9448cfad5365865b5a1ea2, entries=150, sequenceid=318, filesize=11.9 K 2023-04-28 04:56:59,117 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5524ca50728e43bda4cf01d6589461d3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5524ca50728e43bda4cf01d6589461d3 2023-04-28 04:56:59,120 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5524ca50728e43bda4cf01d6589461d3, entries=150, sequenceid=318, filesize=11.9 K 2023-04-28 04:56:59,121 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/6affbab971f44450aedcbd99a278de91 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/6affbab971f44450aedcbd99a278de91 2023-04-28 04:56:59,124 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/6affbab971f44450aedcbd99a278de91, entries=150, sequenceid=318, filesize=11.9 K 2023-04-28 04:56:59,124 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for bf0b022e81bb29fa4ec0f9acedc0e278 in 857ms, sequenceid=318, compaction requested=true 2023-04-28 04:56:59,124 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:56:59,124 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:59,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:59,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:59,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:59,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,201 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:59,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51ec41bd[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:59,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:59,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,211 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:59,211 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:59,211 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:59,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:59,221 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:59,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,221 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,222 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:59,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:59,223 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:59,223 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:59,223 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:59,223 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:59,223 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:59,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,264 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,265 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,317 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:59,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:59,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,317 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,317 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,418 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:59,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:59,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:59,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:59,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:59,421 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:59,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:59,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:59,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:56:59,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-28 04:56:59,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:56:59,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:59,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:56:59,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:59,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:56:59,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:56:59,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657879433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657879433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657879434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657879435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,443 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:59,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:59,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:59,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:59,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:59,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,444 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:59,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:59,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:59,445 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,445 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:59,445 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:59,445 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:59,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:59,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:59,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:59,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:59,455 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:59,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6f35a8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:59,455 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,465 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,465 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,465 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:59,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:59,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:59,466 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:59,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:59,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:59,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:59,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:59,479 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:59,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:59,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:59,479 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:59,479 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:59,485 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,485 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,485 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,486 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,487 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,500 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:59,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,500 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,500 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:59,501 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:59,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657879539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657879539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657879539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657879539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,601 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:59,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:59,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:59,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:59,604 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:59,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:59,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:59,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,604 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:59,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:59,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:59,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:59,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:59,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:59,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:59,612 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,612 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:59,613 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:59,613 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,613 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:59,613 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:59,613 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,613 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:59,613 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:59,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,621 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:59,623 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c958694[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:59,623 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:59,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:59,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:59,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:59,643 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:59,643 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,643 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:59,643 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:59,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:59,645 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:59,645 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:59,645 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:59,645 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:59,645 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:59,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,655 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:59,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:59,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,664 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:59,664 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657879744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:56:59,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657879745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657879744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657879744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,764 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:59,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:59,769 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:59,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:59,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:59,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,779 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,779 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,780 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:59,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:59,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:59,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:59,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:59,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:59,781 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:59,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:59,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:59,782 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:59,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:59,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:59,782 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:59,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:59,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:59,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7302162[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:59,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:59,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:59,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:59,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:59,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:59,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,819 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:59,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:59,821 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:59,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:59,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:59,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:59,821 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:56:59,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,833 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/981fe7e4f05a47ccb7b99a09732b1692 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:56:59,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:56:59,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:56:59,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,843 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:56:59,843 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:56:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:56:59,943 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:56:59,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:56:59,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:56:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:56:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:56:59,946 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:56:59,946 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,946 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:56:59,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:56:59,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:56:59,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:56:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:56:59,959 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,959 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:56:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:56:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:56:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:56:59,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:56:59,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:56:59,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:56:59,963 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,963 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:56:59,963 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:56:59,963 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:56:59,963 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:56:59,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,969 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:56:59,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34626990[Count = 0] remaining members to acquire global barrier 2023-04-28 04:56:59,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:56:59,979 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:56:59,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:56:59,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:56:59,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:56:59,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:56:59,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:56:59,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:56:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:56:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:56:59,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:56:59,992 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:56:59,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:56:59,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:56:59,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:56:59,992 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:00,000 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,000 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,001 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,002 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,011 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:00,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:00,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,012 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:00,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,012 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:00,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,012 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657880049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657880049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657880049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657880050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,112 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:00,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:00,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:00,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:00,115 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:00,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:00,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:00,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:00,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:00,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:00,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:00,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:00,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:00,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:00,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:00,129 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:00,129 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:00,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:00,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:00,129 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:00,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,137 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,138 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,138 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:00,139 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:00,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2034c8c1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:00,139 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:00,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:00,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:00,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:00,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:00,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,158 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:00,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:00,160 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:00,160 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:00,160 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:00,160 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:00,160 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:00,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,170 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:00,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:00,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,180 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:00,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:00,180 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:00,180 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,180 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,180 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/0b8aaf12bc0f43aeb7a527e0cc490678 2023-04-28 04:57:00,280 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:00,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:00,283 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:00,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:00,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:00,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,296 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:00,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:00,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:00,297 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:00,298 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,298 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:00,298 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:00,298 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:00,298 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:00,306 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:00,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20954b75[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:00,308 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:00,308 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,316 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,316 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:00,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:00,317 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:00,317 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:00,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:00,327 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:00,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:00,329 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:00,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:00,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:00,329 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:00,329 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:00,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,338 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,372 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,372 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:00,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:00,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,373 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:00,373 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,373 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:00,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,473 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:00,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:00,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:00,476 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:00,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:00,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:00,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:00,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:00,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:00,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:00,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:00,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:00,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:00,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26284534[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:00,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:00,507 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,516 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,516 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:00,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:00,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:00,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:00,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:00,527 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:00,529 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:00,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:00,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:00,529 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:00,529 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:00,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,539 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:00,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,548 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,548 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:00,548 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,549 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657880553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657880553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657880554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657880556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,649 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:00,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:00,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:00,651 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:00,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:00,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:00,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/f87517f59a144fa7a6aa8dfd22ca364c 2023-04-28 04:57:00,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/981fe7e4f05a47ccb7b99a09732b1692 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/981fe7e4f05a47ccb7b99a09732b1692 2023-04-28 04:57:00,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/981fe7e4f05a47ccb7b99a09732b1692, entries=150, sequenceid=357, filesize=11.9 K 2023-04-28 04:57:00,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/0b8aaf12bc0f43aeb7a527e0cc490678 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/0b8aaf12bc0f43aeb7a527e0cc490678 2023-04-28 04:57:00,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,664 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:00,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:00,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:00,665 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:00,665 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,665 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:00,665 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:00,665 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:00,665 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,665 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:00,665 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:00,666 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:00,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:00,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:00,666 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:00,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/0b8aaf12bc0f43aeb7a527e0cc490678, entries=150, sequenceid=357, filesize=11.9 K 2023-04-28 04:57:00,667 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/f87517f59a144fa7a6aa8dfd22ca364c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f87517f59a144fa7a6aa8dfd22ca364c 2023-04-28 04:57:00,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f87517f59a144fa7a6aa8dfd22ca364c, entries=150, sequenceid=357, filesize=11.9 K 2023-04-28 04:57:00,670 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1241ms, sequenceid=357, compaction requested=true 2023-04-28 04:57:00,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:00,671 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:00,671 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:00,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:57:00,672 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,672 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,672 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/5d257fc4a9994c24a4690b02e109f565, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/37c96da1aa5242dd8feed450ab1c8c15, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7ac4e69a5b9448cfad5365865b5a1ea2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/981fe7e4f05a47ccb7b99a09732b1692] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=48.2 K 2023-04-28 04:57:00,672 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/968d83a1686f42c1ad90cf77027cb2d1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6bd83c40d4504c2797ddab7befd96401, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5524ca50728e43bda4cf01d6589461d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/0b8aaf12bc0f43aeb7a527e0cc490678] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=48.2 K 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5d257fc4a9994c24a4690b02e109f565, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657816984 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 968d83a1686f42c1ad90cf77027cb2d1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657816984 2023-04-28 04:57:00,672 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 37c96da1aa5242dd8feed450ab1c8c15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682657817039 2023-04-28 04:57:00,673 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 6bd83c40d4504c2797ddab7befd96401, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682657817039 2023-04-28 04:57:00,673 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7ac4e69a5b9448cfad5365865b5a1ea2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682657817654 2023-04-28 04:57:00,673 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5524ca50728e43bda4cf01d6589461d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682657817654 2023-04-28 04:57:00,673 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 981fe7e4f05a47ccb7b99a09732b1692, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682657818310 2023-04-28 04:57:00,673 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 0b8aaf12bc0f43aeb7a527e0cc490678, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682657818310 2023-04-28 04:57:00,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,674 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:00,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a19688[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:00,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,685 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:00,685 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:00,685 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:00,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:00,688 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:00,695 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,695 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,695 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,698 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:00,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:00,699 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:00,699 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:00,699 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:00,699 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:00,699 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:00,706 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,706 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,706 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,706 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:00,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,715 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/57ecb098329849e8979d42eb18219feb as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/57ecb098329849e8979d42eb18219feb 2023-04-28 04:57:00,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:00,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:00,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,717 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:00,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,717 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,717 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:00,717 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:00,717 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:00,717 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:00,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:00,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:00,722 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/31b20c9918ba484fb732f2e26f529491 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/31b20c9918ba484fb732f2e26f529491 2023-04-28 04:57:00,724 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into 57ecb098329849e8979d42eb18219feb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:00,724 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:00,724 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=12, startTime=1682657820670; duration=0sec 2023-04-28 04:57:00,724 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:00,724 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:00,727 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:00,727 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:57:00,727 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,727 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/0ae7ce84895c4110a4eeda8b7855e47d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3fd8cfd8646b4e9ab4f7533b87b317fa, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/6affbab971f44450aedcbd99a278de91, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f87517f59a144fa7a6aa8dfd22ca364c] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=48.2 K 2023-04-28 04:57:00,727 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0ae7ce84895c4110a4eeda8b7855e47d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682657816984 2023-04-28 04:57:00,727 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3fd8cfd8646b4e9ab4f7533b87b317fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682657817039 2023-04-28 04:57:00,728 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6affbab971f44450aedcbd99a278de91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682657817654 2023-04-28 04:57:00,728 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f87517f59a144fa7a6aa8dfd22ca364c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682657818310 2023-04-28 04:57:00,730 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into 31b20c9918ba484fb732f2e26f529491(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:00,730 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:00,730 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=12, startTime=1682657820671; duration=0sec 2023-04-28 04:57:00,730 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:00,737 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:00,747 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/cce8a6bcf0384a51a5f76131adcc5d45 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/cce8a6bcf0384a51a5f76131adcc5d45 2023-04-28 04:57:00,750 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into cce8a6bcf0384a51a5f76131adcc5d45(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:00,750 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:00,750 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=12, startTime=1682657820671; duration=0sec 2023-04-28 04:57:00,750 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:00,817 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:00,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:00,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:00,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:00,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:00,820 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:00,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:00,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:00,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:00,832 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:00,832 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:00,832 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:00,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:00,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:00,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:00,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:00,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:00,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:00,834 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:00,834 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:00,834 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:57:00,834 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:00,848 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/4e2f0782c6b545348ffae087767a6952 2023-04-28 04:57:00,856 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/1357f56fca304ef68db607a676501600 2023-04-28 04:57:01,265 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/79198bff571b4ec19c53816c7172f036 2023-04-28 04:57:01,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/4e2f0782c6b545348ffae087767a6952 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4e2f0782c6b545348ffae087767a6952 2023-04-28 04:57:01,272 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4e2f0782c6b545348ffae087767a6952, entries=150, sequenceid=368, filesize=11.9 K 2023-04-28 04:57:01,273 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/1357f56fca304ef68db607a676501600 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1357f56fca304ef68db607a676501600 2023-04-28 04:57:01,276 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1357f56fca304ef68db607a676501600, entries=150, sequenceid=368, filesize=11.9 K 2023-04-28 04:57:01,276 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/79198bff571b4ec19c53816c7172f036 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79198bff571b4ec19c53816c7172f036 2023-04-28 04:57:01,281 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79198bff571b4ec19c53816c7172f036, entries=150, sequenceid=368, filesize=11.9 K 2023-04-28 04:57:01,282 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for bf0b022e81bb29fa4ec0f9acedc0e278 in 448ms, sequenceid=368, compaction requested=false 2023-04-28 04:57:01,282 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:01,282 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:01,282 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:01,282 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:01,282 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,282 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:01,282 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:01,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:01,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75524bd[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:01,381 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:01,381 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:01,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:01,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:01,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:01,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:01,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:01,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:01,403 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:01,403 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:01,403 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:01,403 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:01,403 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:01,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:01,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,411 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:01,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:01,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:01,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:01,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:01,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:01,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:01,422 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:01,422 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:01,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:01,522 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:01,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:01,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:01,525 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:01,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:01,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:01,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:01,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,537 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:01,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:01,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:01,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:01,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:01,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:01,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:01,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:01,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:01,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:01,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:57:01,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:57:01,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:01,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:57:01,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:01,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:57:01,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:01,576 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,576 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/42ac03117098459c8e23cf269395092c 2023-04-28 04:57:01,577 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,577 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:01,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:01,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1119c1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:01,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657821572, max timestamp=1682657821583Num uniques 859; 2023-04-28 04:57:01,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:01,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657821572, max timestamp=1682657821583Num uniques 859; 2023-04-28 04:57:01,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:01,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657821572, max timestamp=1682657821583Num uniques 859; 2023-04-28 04:57:01,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:01,601 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:01,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:01,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:01,601 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657881615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657881615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657881615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657881616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:01,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:01,655 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:01,655 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:01,655 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:01,655 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:01,655 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657881719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657881719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657881719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657881719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,744 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,744 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,744 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:01,745 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,746 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:01,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:01,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:01,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:01,769 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:01,769 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,769 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,769 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:01,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:01,870 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:01,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:01,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:01,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:01,872 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:01,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:01,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:01,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:01,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:01,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:01,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:01,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:01,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:01,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:01,886 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:01,886 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:01,886 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:01,886 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:01,886 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:01,886 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:01,886 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:01,887 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:01,887 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:01,887 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:01,887 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:01,887 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,887 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:01,887 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:01,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,896 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:01,897 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:01,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f4fcb2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:01,897 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,906 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:01,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:01,906 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,906 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:01,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:01,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:01,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:01,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:01,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:01,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:01,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:01,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:01,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:01,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:01,919 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:01,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:01,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:01,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:01,919 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:01,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657881921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657881921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657881922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657881922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:01,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/81d6ec159e1644389dccec57ec85ab72 2023-04-28 04:57:02,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,034 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:02,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:02,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:02,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:02,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:02,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:02,043 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,043 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:02,143 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:02,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:02,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:02,146 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:02,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:02,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:02,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:02,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,158 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,158 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:02,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:02,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,159 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:02,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:02,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:02,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:02,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:02,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:02,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:02,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ec22d2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:02,170 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:02,170 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,179 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:02,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:02,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:02,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:02,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:02,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:02,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:02,192 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:02,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:02,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:02,192 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:02,192 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:02,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,202 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:02,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:02,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:02,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:02,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:02,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:02,211 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:02,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:02,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,211 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:02,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:02,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:02,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657882224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657882225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657882225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657882225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,312 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:02,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:02,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:02,314 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:02,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:02,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:02,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:02,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,327 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:02,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:02,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:02,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:02,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:02,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:02,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:02,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:02,328 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:02,329 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:02,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:02,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ab2823e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:02,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:02,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,377 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:02,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:02,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:02,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:02,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/d85fa6a4d9704f62bb74f6cd2246a06a 2023-04-28 04:57:02,399 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:02,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,399 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/42ac03117098459c8e23cf269395092c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/42ac03117098459c8e23cf269395092c 2023-04-28 04:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:02,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:02,402 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:02,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:02,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:02,402 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:02,402 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:02,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/42ac03117098459c8e23cf269395092c, entries=100, sequenceid=379, filesize=9.5 K 2023-04-28 04:57:02,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/81d6ec159e1644389dccec57ec85ab72 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/81d6ec159e1644389dccec57ec85ab72 2023-04-28 04:57:02,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/81d6ec159e1644389dccec57ec85ab72, entries=100, sequenceid=379, filesize=9.5 K 2023-04-28 04:57:02,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/d85fa6a4d9704f62bb74f6cd2246a06a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/d85fa6a4d9704f62bb74f6cd2246a06a 2023-04-28 04:57:02,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:02,411 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:02,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:02,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:02,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:02,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:02,422 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:02,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/d85fa6a4d9704f62bb74f6cd2246a06a, entries=100, sequenceid=379, filesize=9.5 K 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:02,422 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:02,422 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,423 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:02,423 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bf0b022e81bb29fa4ec0f9acedc0e278 in 857ms, sequenceid=379, compaction requested=true 2023-04-28 04:57:02,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:02,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:02,423 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:02,423 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:02,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:02,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:02,424 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:02,424 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:02,424 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/A is initiating minor compaction (all files) 2023-04-28 04:57:02,424 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/B is initiating minor compaction (all files) 2023-04-28 04:57:02,424 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/A in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,424 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/B in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,424 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/57ecb098329849e8979d42eb18219feb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4e2f0782c6b545348ffae087767a6952, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/42ac03117098459c8e23cf269395092c] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=34.0 K 2023-04-28 04:57:02,424 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/31b20c9918ba484fb732f2e26f529491, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1357f56fca304ef68db607a676501600, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/81d6ec159e1644389dccec57ec85ab72] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=34.0 K 2023-04-28 04:57:02,425 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 57ecb098329849e8979d42eb18219feb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682657818310 2023-04-28 04:57:02,425 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 31b20c9918ba484fb732f2e26f529491, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682657818310 2023-04-28 04:57:02,425 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4e2f0782c6b545348ffae087767a6952, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682657819432 2023-04-28 04:57:02,426 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 42ac03117098459c8e23cf269395092c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682657821565 2023-04-28 04:57:02,426 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1357f56fca304ef68db607a676501600, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682657819432 2023-04-28 04:57:02,426 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 81d6ec159e1644389dccec57ec85ab72, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682657821565 2023-04-28 04:57:02,433 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#A#compaction#245 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:02,434 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#B#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:02,446 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/e8b36e3fced9451e8f8b443b3045280f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/e8b36e3fced9451e8f8b443b3045280f 2023-04-28 04:57:02,448 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/4c226e4832c24665a450ebf25181da63 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/4c226e4832c24665a450ebf25181da63 2023-04-28 04:57:02,450 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/A of bf0b022e81bb29fa4ec0f9acedc0e278 into e8b36e3fced9451e8f8b443b3045280f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:02,450 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:02,450 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/A, priority=13, startTime=1682657822423; duration=0sec 2023-04-28 04:57:02,451 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:02,451 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:02,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:02,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): bf0b022e81bb29fa4ec0f9acedc0e278/C is initiating minor compaction (all files) 2023-04-28 04:57:02,452 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf0b022e81bb29fa4ec0f9acedc0e278/C in TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,452 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/cce8a6bcf0384a51a5f76131adcc5d45, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79198bff571b4ec19c53816c7172f036, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/d85fa6a4d9704f62bb74f6cd2246a06a] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp, totalSize=34.0 K 2023-04-28 04:57:02,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting cce8a6bcf0384a51a5f76131adcc5d45, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682657818310 2023-04-28 04:57:02,452 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/B of bf0b022e81bb29fa4ec0f9acedc0e278 into 4c226e4832c24665a450ebf25181da63(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:02,452 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:02,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 79198bff571b4ec19c53816c7172f036, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682657819432 2023-04-28 04:57:02,452 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/B, priority=13, startTime=1682657822423; duration=0sec 2023-04-28 04:57:02,453 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:02,453 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d85fa6a4d9704f62bb74f6cd2246a06a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682657821565 2023-04-28 04:57:02,457 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf0b022e81bb29fa4ec0f9acedc0e278#C#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:02,463 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/4ea827307fdd4a778ce111148c2c9788 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/4ea827307fdd4a778ce111148c2c9788 2023-04-28 04:57:02,467 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf0b022e81bb29fa4ec0f9acedc0e278/C of bf0b022e81bb29fa4ec0f9acedc0e278 into 4ea827307fdd4a778ce111148c2c9788(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:02,467 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:02,467 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278., storeName=bf0b022e81bb29fa4ec0f9acedc0e278/C, priority=13, startTime=1682657822423; duration=0sec 2023-04-28 04:57:02,467 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:02,523 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:02,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:02,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:02,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:02,525 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:02,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:02,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:02,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:02,526 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,526 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,537 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:02,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:02,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:02,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:02,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:02,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:02,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:02,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:02,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:02,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:02,539 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:02,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:02,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:02,539 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-28 04:57:02,540 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:57:02,540 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:02,540 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:57:02,540 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:02,540 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:57:02,540 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:02,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:02,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657882734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657882753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657882753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657882753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657882854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657882856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657882856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657882856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:02,947 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/b003613abab74f1492ed139ab623085f 2023-04-28 04:57:02,957 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/d8063c147691461c9fd7e777b274362f 2023-04-28 04:57:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657883058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657883058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657883058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657883059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657883360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657883360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657883361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657883362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,368 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/117d06ba926249fc86a16f383b123c37 2023-04-28 04:57:03,371 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/b003613abab74f1492ed139ab623085f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/b003613abab74f1492ed139ab623085f 2023-04-28 04:57:03,374 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/b003613abab74f1492ed139ab623085f, entries=150, sequenceid=418, filesize=11.9 K 2023-04-28 04:57:03,375 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/d8063c147691461c9fd7e777b274362f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/d8063c147691461c9fd7e777b274362f 2023-04-28 04:57:03,378 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/d8063c147691461c9fd7e777b274362f, entries=150, sequenceid=418, filesize=11.9 K 2023-04-28 04:57:03,379 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/117d06ba926249fc86a16f383b123c37 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/117d06ba926249fc86a16f383b123c37 2023-04-28 04:57:03,382 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/117d06ba926249fc86a16f383b123c37, entries=150, sequenceid=418, filesize=11.9 K 2023-04-28 04:57:03,383 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for bf0b022e81bb29fa4ec0f9acedc0e278 in 844ms, sequenceid=418, compaction requested=false 2023-04-28 04:57:03,383 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:03,383 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:03,383 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:03,383 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:03,383 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,383 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:03,383 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:03,411 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:03,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe855b8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:03,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:03,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:03,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:03,422 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:03,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:03,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:03,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:03,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:03,434 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:03,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:03,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:03,434 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:03,434 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:03,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:03,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:03,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:03,444 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:03,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:03,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:03,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:03,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:03,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:03,453 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,453 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:03,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:03,454 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:03,554 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:03,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:03,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:03,556 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:03,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:03,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:03,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:03,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:03,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:03,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:03,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:03,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:03,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:03,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:03,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:03,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:03,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:03,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:03,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:03,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:03,569 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. started... 2023-04-28 04:57:03,570 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:57:03,570 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:03,576 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/15de5b55f02c417d8704d265318ab5e0 2023-04-28 04:57:03,585 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/16aaab24784846aa947707c6c43b2c8b 2023-04-28 04:57:03,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:03,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. as already flushing 2023-04-28 04:57:03,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657823874, max timestamp=1682657823877Num uniques 859; 2023-04-28 04:57:03,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:57:03,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657823874, max timestamp=1682657823877Num uniques 859; 2023-04-28 04:57:03,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:57:03,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657823874, max timestamp=1682657823877Num uniques 859; 2023-04-28 04:57:03,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:57:03,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657883892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657883893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657883894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657883894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,951 DEBUG [Thread-1423] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0993cc7d to 127.0.0.1:52932 2023-04-28 04:57:03,951 DEBUG [Thread-1423] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:03,951 DEBUG [Thread-1427] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1656b283 to 127.0.0.1:52932 2023-04-28 04:57:03,951 DEBUG [Thread-1427] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:03,953 DEBUG [Thread-1419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57d25552 to 127.0.0.1:52932 2023-04-28 04:57:03,953 DEBUG [Thread-1419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:03,953 DEBUG [Thread-1425] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69b9b3fd to 127.0.0.1:52932 2023-04-28 04:57:03,953 DEBUG [Thread-1425] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:03,953 DEBUG [Thread-1421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13695730 to 127.0.0.1:52932 2023-04-28 04:57:03,953 DEBUG [Thread-1421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:03,994 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/82468f011a534008b4fcece0e97a964d 2023-04-28 04:57:03,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58494 deadline: 1682657883996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58506 deadline: 1682657883996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58498 deadline: 1682657883997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:03,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:03,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:58496 deadline: 1682657883998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,000 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/15de5b55f02c417d8704d265318ab5e0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/15de5b55f02c417d8704d265318ab5e0 2023-04-28 04:57:04,006 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/15de5b55f02c417d8704d265318ab5e0, entries=150, sequenceid=428, filesize=11.9 K 2023-04-28 04:57:04,007 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/16aaab24784846aa947707c6c43b2c8b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/16aaab24784846aa947707c6c43b2c8b 2023-04-28 04:57:04,011 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/16aaab24784846aa947707c6c43b2c8b, entries=150, sequenceid=428, filesize=11.9 K 2023-04-28 04:57:04,011 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/82468f011a534008b4fcece0e97a964d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/82468f011a534008b4fcece0e97a964d 2023-04-28 04:57:04,014 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/82468f011a534008b4fcece0e97a964d, entries=150, sequenceid=428, filesize=11.9 K 2023-04-28 04:57:04,015 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bf0b022e81bb29fa4ec0f9acedc0e278 in 445ms, sequenceid=428, compaction requested=true 2023-04-28 04:57:04,015 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:04,015 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:04,015 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:04,015 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:04,015 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,015 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:04,015 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:04,021 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:04,022 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,022 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:04,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:04,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b7219b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:04,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:04,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,032 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:04,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:04,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:04,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:04,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:04,043 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:04,043 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:04,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:04,045 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:04,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:04,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:04,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:04,045 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:04,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:04,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:04,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:04,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:04,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:04,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:04,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:04,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:04,064 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:04,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:04,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,064 DEBUG [Thread-1418] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:04,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:04,064 DEBUG [Thread-1418] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:04,165 DEBUG [Thread-1418] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:04,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:04,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:04,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-28 04:57:04,199 DEBUG [Thread-1412] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e3efec6 to 127.0.0.1:52932 2023-04-28 04:57:04,199 DEBUG [Thread-1412] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:04,199 DEBUG [Thread-1410] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43184c57 to 127.0.0.1:52932 2023-04-28 04:57:04,200 DEBUG [Thread-1410] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:04,200 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:57:04,200 DEBUG [Thread-1414] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b7f716c to 127.0.0.1:52932 2023-04-28 04:57:04,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:57:04,201 DEBUG [Thread-1414] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:04,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:57:04,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:57:04,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:57:04,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:57:04,203 DEBUG [Thread-1416] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ade356f to 127.0.0.1:52932 2023-04-28 04:57:04,203 DEBUG [Thread-1416] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:04,417 DEBUG [Thread-1408] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29a548ac to 127.0.0.1:52932 2023-04-28 04:57:04,417 DEBUG [Thread-1408] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 98 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 6268 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5796 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5984 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 6320 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5770 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 04:57:04,417 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:57:04,417 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12394d27 to 127.0.0.1:52932 2023-04-28 04:57:04,417 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:04,418 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 04:57:04,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$11(2418): Client=jenkins//136.243.104.168 disable TestAcidGuarantees 2023-04-28 04:57:04,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:04,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:04,421 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657824421"}]},"ts":"1682657824421"} 2023-04-28 04:57:04,422 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 04:57:04,443 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 04:57:04,444 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, UNASSIGN}] 2023-04-28 04:57:04,445 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, UNASSIGN 2023-04-28 04:57:04,446 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=bf0b022e81bb29fa4ec0f9acedc0e278, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:04,446 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657824445"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657824445"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657824445"}]},"ts":"1682657824445"} 2023-04-28 04:57:04,447 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:57:04,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:04,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:04,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing bf0b022e81bb29fa4ec0f9acedc0e278, disabling compactions & flushes 2023-04-28 04:57:04,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:04,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/de0391a2ae94423d9d8c7315039c8680 2023-04-28 04:57:04,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:05,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:05,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5877710b6f954e5d8a6574be3a4761af 2023-04-28 04:57:05,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/961154f658614a20acc25561abcdd4dc 2023-04-28 04:57:05,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/de0391a2ae94423d9d8c7315039c8680 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/de0391a2ae94423d9d8c7315039c8680 2023-04-28 04:57:05,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/de0391a2ae94423d9d8c7315039c8680, entries=150, sequenceid=468, filesize=11.9 K 2023-04-28 04:57:05,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/5877710b6f954e5d8a6574be3a4761af as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5877710b6f954e5d8a6574be3a4761af 2023-04-28 04:57:05,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5877710b6f954e5d8a6574be3a4761af, entries=150, sequenceid=468, filesize=11.9 K 2023-04-28 04:57:05,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/961154f658614a20acc25561abcdd4dc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/961154f658614a20acc25561abcdd4dc 2023-04-28 04:57:05,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/961154f658614a20acc25561abcdd4dc, entries=150, sequenceid=468, filesize=11.9 K 2023-04-28 04:57:05,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~445.41 KB/456096, currentSize=13.42 KB/13740 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1255ms, sequenceid=468, compaction requested=true 2023-04-28 04:57:05,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:05,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:05,454 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. because compaction request was cancelled 2023-04-28 04:57:05,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:05,454 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. because compaction request was cancelled 2023-04-28 04:57:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:05,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. after waiting 0 ms 2023-04-28 04:57:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:05,454 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. because compaction request was cancelled 2023-04-28 04:57:05,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:05,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing bf0b022e81bb29fa4ec0f9acedc0e278 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-28 04:57:05,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=A 2023-04-28 04:57:05,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:05,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=B 2023-04-28 04:57:05,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:05,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf0b022e81bb29fa4ec0f9acedc0e278, store=C 2023-04-28 04:57:05,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:05,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/f0c3108d61cd4d2f992a7f5a99f55a7e 2023-04-28 04:57:06,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/41f5143b2aa64319a274bf815a404d98 2023-04-28 04:57:06,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:06,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/2d1e258e0f524abc9ee1416c0631f6b8 2023-04-28 04:57:06,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/A/f0c3108d61cd4d2f992a7f5a99f55a7e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/f0c3108d61cd4d2f992a7f5a99f55a7e 2023-04-28 04:57:06,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/f0c3108d61cd4d2f992a7f5a99f55a7e, entries=100, sequenceid=473, filesize=9.5 K 2023-04-28 04:57:06,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/B/41f5143b2aa64319a274bf815a404d98 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f5143b2aa64319a274bf815a404d98 2023-04-28 04:57:06,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f5143b2aa64319a274bf815a404d98, entries=100, sequenceid=473, filesize=9.5 K 2023-04-28 04:57:06,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/.tmp/C/2d1e258e0f524abc9ee1416c0631f6b8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2d1e258e0f524abc9ee1416c0631f6b8 2023-04-28 04:57:06,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2d1e258e0f524abc9ee1416c0631f6b8, entries=100, sequenceid=473, filesize=9.5 K 2023-04-28 04:57:06,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for bf0b022e81bb29fa4ec0f9acedc0e278 in 1241ms, sequenceid=473, compaction requested=true 2023-04-28 04:57:06,712 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/525a51f1006c4792a642b87e7971a7c1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/efd97a439a694a629b946cd5726dc83b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3a6d9dd852fa413aaf97eb3f23bea0b9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6158b64652b04f3c917d6a8c646d8f58, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/d5b497cd3d9c455db808e9308669936c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/55ee5f3d0d3948928465c55df69a2495, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/48e202533de845078553984241c445e2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3f30a903cba2448cac29bbf4ddaae50f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4947e23b93984e0b90da9c4eb087b560, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/9d799c3413da48ff88a5d670fa32a8ee, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/65b1ad830b3a4387a32993457b123138, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7590837770ee4fc4ad064ec90100f055, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/121bd058a3bb41da9f96956050b01681, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6d027f5073794d6e9e1559b4f57e0e7c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/05ad2f973da941e3835c84f11b086098, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/91a235d7071e45d99762fbf3288a8e19, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/eed7161755184ad88da593eff32f3e34, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/5d257fc4a9994c24a4690b02e109f565, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/37c96da1aa5242dd8feed450ab1c8c15, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7ac4e69a5b9448cfad5365865b5a1ea2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/57ecb098329849e8979d42eb18219feb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/981fe7e4f05a47ccb7b99a09732b1692, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4e2f0782c6b545348ffae087767a6952, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/42ac03117098459c8e23cf269395092c] to archive 2023-04-28 04:57:06,713 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:06,714 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/525a51f1006c4792a642b87e7971a7c1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/525a51f1006c4792a642b87e7971a7c1 2023-04-28 04:57:06,715 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/efd97a439a694a629b946cd5726dc83b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/efd97a439a694a629b946cd5726dc83b 2023-04-28 04:57:06,716 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3a6d9dd852fa413aaf97eb3f23bea0b9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3a6d9dd852fa413aaf97eb3f23bea0b9 2023-04-28 04:57:06,717 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6158b64652b04f3c917d6a8c646d8f58 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6158b64652b04f3c917d6a8c646d8f58 2023-04-28 04:57:06,718 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/d5b497cd3d9c455db808e9308669936c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/d5b497cd3d9c455db808e9308669936c 2023-04-28 04:57:06,719 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/55ee5f3d0d3948928465c55df69a2495 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/55ee5f3d0d3948928465c55df69a2495 2023-04-28 04:57:06,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/48e202533de845078553984241c445e2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/48e202533de845078553984241c445e2 2023-04-28 04:57:06,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3f30a903cba2448cac29bbf4ddaae50f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/3f30a903cba2448cac29bbf4ddaae50f 2023-04-28 04:57:06,722 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4947e23b93984e0b90da9c4eb087b560 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4947e23b93984e0b90da9c4eb087b560 2023-04-28 04:57:06,723 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/9d799c3413da48ff88a5d670fa32a8ee to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/9d799c3413da48ff88a5d670fa32a8ee 2023-04-28 04:57:06,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/65b1ad830b3a4387a32993457b123138 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/65b1ad830b3a4387a32993457b123138 2023-04-28 04:57:06,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7590837770ee4fc4ad064ec90100f055 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7590837770ee4fc4ad064ec90100f055 2023-04-28 04:57:06,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/121bd058a3bb41da9f96956050b01681 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/121bd058a3bb41da9f96956050b01681 2023-04-28 04:57:06,727 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6d027f5073794d6e9e1559b4f57e0e7c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/6d027f5073794d6e9e1559b4f57e0e7c 2023-04-28 04:57:06,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/05ad2f973da941e3835c84f11b086098 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/05ad2f973da941e3835c84f11b086098 2023-04-28 04:57:06,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/91a235d7071e45d99762fbf3288a8e19 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/91a235d7071e45d99762fbf3288a8e19 2023-04-28 04:57:06,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/eed7161755184ad88da593eff32f3e34 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/eed7161755184ad88da593eff32f3e34 2023-04-28 04:57:06,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/5d257fc4a9994c24a4690b02e109f565 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/5d257fc4a9994c24a4690b02e109f565 2023-04-28 04:57:06,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/37c96da1aa5242dd8feed450ab1c8c15 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/37c96da1aa5242dd8feed450ab1c8c15 2023-04-28 04:57:06,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7ac4e69a5b9448cfad5365865b5a1ea2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/7ac4e69a5b9448cfad5365865b5a1ea2 2023-04-28 04:57:06,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/57ecb098329849e8979d42eb18219feb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/57ecb098329849e8979d42eb18219feb 2023-04-28 04:57:06,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/981fe7e4f05a47ccb7b99a09732b1692 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/981fe7e4f05a47ccb7b99a09732b1692 2023-04-28 04:57:06,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4e2f0782c6b545348ffae087767a6952 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/4e2f0782c6b545348ffae087767a6952 2023-04-28 04:57:06,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/42ac03117098459c8e23cf269395092c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/42ac03117098459c8e23cf269395092c 2023-04-28 04:57:06,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d683fa5ea334109a172a56203e6392a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/cb8c078b26c94b279e627470264c430c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c0fd289d1d184e079137a6129373dcd9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/7ba61777016e42a69438131effb16c12, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/03152deba7e74dc18ee2aafcfdd2fa41, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f8ca860f164070aed3e1ea42834907, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6426b121ec254b349275181bf2afd4c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1ee95c570fbe471c9b2956835565420c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/12f3ecc0bb394ef59c4454bdf254e0e1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/8764e0324cc94427badf34f1a505e4b1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/a01807738df74d4f9bc64e8db77e44e6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/ff8f88a9bf004336abc51fb9d506fd4f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c9b622312af34340b288f3e9c4121b74, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6c381c65f60046758ccd0119d756738d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/b6a25ccadbec4f0eb66c26aec557ac4a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/968d83a1686f42c1ad90cf77027cb2d1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/530078fd48414282bff3fa976560e857, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6bd83c40d4504c2797ddab7befd96401, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5524ca50728e43bda4cf01d6589461d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/31b20c9918ba484fb732f2e26f529491, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/0b8aaf12bc0f43aeb7a527e0cc490678, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1357f56fca304ef68db607a676501600, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/81d6ec159e1644389dccec57ec85ab72] to archive 2023-04-28 04:57:06,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:06,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d683fa5ea334109a172a56203e6392a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d683fa5ea334109a172a56203e6392a 2023-04-28 04:57:06,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/cb8c078b26c94b279e627470264c430c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/cb8c078b26c94b279e627470264c430c 2023-04-28 04:57:06,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c0fd289d1d184e079137a6129373dcd9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c0fd289d1d184e079137a6129373dcd9 2023-04-28 04:57:06,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/7ba61777016e42a69438131effb16c12 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/7ba61777016e42a69438131effb16c12 2023-04-28 04:57:06,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/03152deba7e74dc18ee2aafcfdd2fa41 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/03152deba7e74dc18ee2aafcfdd2fa41 2023-04-28 04:57:06,748 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f8ca860f164070aed3e1ea42834907 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f8ca860f164070aed3e1ea42834907 2023-04-28 04:57:06,749 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6426b121ec254b349275181bf2afd4c0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6426b121ec254b349275181bf2afd4c0 2023-04-28 04:57:06,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5d1a5f5432b44a37b8b05d8e6cfb1cb3 2023-04-28 04:57:06,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1ee95c570fbe471c9b2956835565420c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1ee95c570fbe471c9b2956835565420c 2023-04-28 04:57:06,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/12f3ecc0bb394ef59c4454bdf254e0e1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/12f3ecc0bb394ef59c4454bdf254e0e1 2023-04-28 04:57:06,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/8764e0324cc94427badf34f1a505e4b1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/8764e0324cc94427badf34f1a505e4b1 2023-04-28 04:57:06,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/a01807738df74d4f9bc64e8db77e44e6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/a01807738df74d4f9bc64e8db77e44e6 2023-04-28 04:57:06,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/ff8f88a9bf004336abc51fb9d506fd4f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/ff8f88a9bf004336abc51fb9d506fd4f 2023-04-28 04:57:06,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c9b622312af34340b288f3e9c4121b74 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/c9b622312af34340b288f3e9c4121b74 2023-04-28 04:57:06,756 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6c381c65f60046758ccd0119d756738d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6c381c65f60046758ccd0119d756738d 2023-04-28 04:57:06,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/b6a25ccadbec4f0eb66c26aec557ac4a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/b6a25ccadbec4f0eb66c26aec557ac4a 2023-04-28 04:57:06,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/968d83a1686f42c1ad90cf77027cb2d1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/968d83a1686f42c1ad90cf77027cb2d1 2023-04-28 04:57:06,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/530078fd48414282bff3fa976560e857 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/530078fd48414282bff3fa976560e857 2023-04-28 04:57:06,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6bd83c40d4504c2797ddab7befd96401 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/6bd83c40d4504c2797ddab7befd96401 2023-04-28 04:57:06,760 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5524ca50728e43bda4cf01d6589461d3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5524ca50728e43bda4cf01d6589461d3 2023-04-28 04:57:06,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/31b20c9918ba484fb732f2e26f529491 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/31b20c9918ba484fb732f2e26f529491 2023-04-28 04:57:06,762 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/0b8aaf12bc0f43aeb7a527e0cc490678 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/0b8aaf12bc0f43aeb7a527e0cc490678 2023-04-28 04:57:06,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1357f56fca304ef68db607a676501600 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/1357f56fca304ef68db607a676501600 2023-04-28 04:57:06,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/81d6ec159e1644389dccec57ec85ab72 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/81d6ec159e1644389dccec57ec85ab72 2023-04-28 04:57:06,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/e0f76405a6714bec92445687eb837d18, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/660d3eb790354e24b3330e310acb7a68, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1af301275a44886ad0a06b32d9e3d48, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/48a967cc51fd40acbb22ae67e2d43e5e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/5dc4e995f26e4e71a3c092da99d35384, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/1ea0fa0d0f7444cd94e480784766eb62, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2f79bd25a48c46a381dda654fd8ce026, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/12d7e64b66ce481da16753ec24dffb8a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79659cc293604eb7b9bafc40e78307a2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/26aba9573eff4e2f8b9f4787e40efdc2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2b870a680d4c47988617010187443acd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b5c861ecdf4847049ca9da96f4a5658c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/c99a94a55ecf463895ec147ae8aa20b8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3f616a1b22334edf963dbdee08bbf191, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1bfbec945a14d4f9fe0588d59151a78, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f1ec80ceb4da47edb0a37c8a16068da6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/0ae7ce84895c4110a4eeda8b7855e47d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/09f925c65a3048549ad7adb55993f98a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3fd8cfd8646b4e9ab4f7533b87b317fa, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/6affbab971f44450aedcbd99a278de91, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/cce8a6bcf0384a51a5f76131adcc5d45, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f87517f59a144fa7a6aa8dfd22ca364c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79198bff571b4ec19c53816c7172f036, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/d85fa6a4d9704f62bb74f6cd2246a06a] to archive 2023-04-28 04:57:06,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:06,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/e0f76405a6714bec92445687eb837d18 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/e0f76405a6714bec92445687eb837d18 2023-04-28 04:57:06,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/660d3eb790354e24b3330e310acb7a68 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/660d3eb790354e24b3330e310acb7a68 2023-04-28 04:57:06,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1af301275a44886ad0a06b32d9e3d48 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1af301275a44886ad0a06b32d9e3d48 2023-04-28 04:57:06,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/48a967cc51fd40acbb22ae67e2d43e5e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/48a967cc51fd40acbb22ae67e2d43e5e 2023-04-28 04:57:06,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/5dc4e995f26e4e71a3c092da99d35384 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/5dc4e995f26e4e71a3c092da99d35384 2023-04-28 04:57:06,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/1ea0fa0d0f7444cd94e480784766eb62 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/1ea0fa0d0f7444cd94e480784766eb62 2023-04-28 04:57:06,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2f79bd25a48c46a381dda654fd8ce026 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2f79bd25a48c46a381dda654fd8ce026 2023-04-28 04:57:06,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/12d7e64b66ce481da16753ec24dffb8a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/12d7e64b66ce481da16753ec24dffb8a 2023-04-28 04:57:06,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79659cc293604eb7b9bafc40e78307a2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79659cc293604eb7b9bafc40e78307a2 2023-04-28 04:57:06,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/26aba9573eff4e2f8b9f4787e40efdc2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/26aba9573eff4e2f8b9f4787e40efdc2 2023-04-28 04:57:06,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2b870a680d4c47988617010187443acd to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2b870a680d4c47988617010187443acd 2023-04-28 04:57:06,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b5c861ecdf4847049ca9da96f4a5658c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b5c861ecdf4847049ca9da96f4a5658c 2023-04-28 04:57:06,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/c99a94a55ecf463895ec147ae8aa20b8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/c99a94a55ecf463895ec147ae8aa20b8 2023-04-28 04:57:06,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3f616a1b22334edf963dbdee08bbf191 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3f616a1b22334edf963dbdee08bbf191 2023-04-28 04:57:06,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1bfbec945a14d4f9fe0588d59151a78 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/b1bfbec945a14d4f9fe0588d59151a78 2023-04-28 04:57:06,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f1ec80ceb4da47edb0a37c8a16068da6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f1ec80ceb4da47edb0a37c8a16068da6 2023-04-28 04:57:06,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/0ae7ce84895c4110a4eeda8b7855e47d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/0ae7ce84895c4110a4eeda8b7855e47d 2023-04-28 04:57:06,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/09f925c65a3048549ad7adb55993f98a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/09f925c65a3048549ad7adb55993f98a 2023-04-28 04:57:06,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3fd8cfd8646b4e9ab4f7533b87b317fa to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/3fd8cfd8646b4e9ab4f7533b87b317fa 2023-04-28 04:57:06,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/6affbab971f44450aedcbd99a278de91 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/6affbab971f44450aedcbd99a278de91 2023-04-28 04:57:06,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/cce8a6bcf0384a51a5f76131adcc5d45 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/cce8a6bcf0384a51a5f76131adcc5d45 2023-04-28 04:57:06,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f87517f59a144fa7a6aa8dfd22ca364c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/f87517f59a144fa7a6aa8dfd22ca364c 2023-04-28 04:57:06,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79198bff571b4ec19c53816c7172f036 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/79198bff571b4ec19c53816c7172f036 2023-04-28 04:57:06,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/d85fa6a4d9704f62bb74f6cd2246a06a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/d85fa6a4d9704f62bb74f6cd2246a06a 2023-04-28 04:57:06,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/recovered.edits/476.seqid, newMaxSeqId=476, maxSeqId=1 2023-04-28 04:57:06,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278. 2023-04-28 04:57:06,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for bf0b022e81bb29fa4ec0f9acedc0e278: 2023-04-28 04:57:06,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:06,791 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=bf0b022e81bb29fa4ec0f9acedc0e278, regionState=CLOSED 2023-04-28 04:57:06,792 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657826791"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657826791"}]},"ts":"1682657826791"} 2023-04-28 04:57:06,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-28 04:57:06,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure bf0b022e81bb29fa4ec0f9acedc0e278, server=jenkins-hbase12.apache.org,37663,1682657733313 in 2.3450 sec 2023-04-28 04:57:06,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-28 04:57:06,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf0b022e81bb29fa4ec0f9acedc0e278, UNASSIGN in 2.3510 sec 2023-04-28 04:57:06,795 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657826795"}]},"ts":"1682657826795"} 2023-04-28 04:57:06,796 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 04:57:06,838 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 04:57:06,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4210 sec 2023-04-28 04:57:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 04:57:08,527 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-28 04:57:08,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$5(2228): Client=jenkins//136.243.104.168 delete TestAcidGuarantees 2023-04-28 04:57:08,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,529 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 04:57:08,530 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,533 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:08,535 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/recovered.edits] 2023-04-28 04:57:08,538 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/15de5b55f02c417d8704d265318ab5e0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/15de5b55f02c417d8704d265318ab5e0 2023-04-28 04:57:08,539 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/b003613abab74f1492ed139ab623085f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/b003613abab74f1492ed139ab623085f 2023-04-28 04:57:08,541 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/de0391a2ae94423d9d8c7315039c8680 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/de0391a2ae94423d9d8c7315039c8680 2023-04-28 04:57:08,542 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/e8b36e3fced9451e8f8b443b3045280f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/e8b36e3fced9451e8f8b443b3045280f 2023-04-28 04:57:08,543 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/f0c3108d61cd4d2f992a7f5a99f55a7e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/A/f0c3108d61cd4d2f992a7f5a99f55a7e 2023-04-28 04:57:08,545 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/16aaab24784846aa947707c6c43b2c8b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/16aaab24784846aa947707c6c43b2c8b 2023-04-28 04:57:08,546 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f5143b2aa64319a274bf815a404d98 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/41f5143b2aa64319a274bf815a404d98 2023-04-28 04:57:08,547 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/4c226e4832c24665a450ebf25181da63 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/4c226e4832c24665a450ebf25181da63 2023-04-28 04:57:08,548 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5877710b6f954e5d8a6574be3a4761af to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/5877710b6f954e5d8a6574be3a4761af 2023-04-28 04:57:08,549 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/d8063c147691461c9fd7e777b274362f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/B/d8063c147691461c9fd7e777b274362f 2023-04-28 04:57:08,551 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/117d06ba926249fc86a16f383b123c37 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/117d06ba926249fc86a16f383b123c37 2023-04-28 04:57:08,552 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2d1e258e0f524abc9ee1416c0631f6b8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/2d1e258e0f524abc9ee1416c0631f6b8 2023-04-28 04:57:08,553 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/4ea827307fdd4a778ce111148c2c9788 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/4ea827307fdd4a778ce111148c2c9788 2023-04-28 04:57:08,554 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/82468f011a534008b4fcece0e97a964d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/82468f011a534008b4fcece0e97a964d 2023-04-28 04:57:08,555 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/961154f658614a20acc25561abcdd4dc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/C/961154f658614a20acc25561abcdd4dc 2023-04-28 04:57:08,557 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/recovered.edits/476.seqid to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278/recovered.edits/476.seqid 2023-04-28 04:57:08,558 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/bf0b022e81bb29fa4ec0f9acedc0e278 2023-04-28 04:57:08,558 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:57:08,560 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,561 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 04:57:08,563 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 04:57:08,564 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,564 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 04:57:08,564 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682657828564"}]},"ts":"9223372036854775807"} 2023-04-28 04:57:08,566 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 04:57:08,566 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bf0b022e81bb29fa4ec0f9acedc0e278, NAME => 'TestAcidGuarantees,,1682657801542.bf0b022e81bb29fa4ec0f9acedc0e278.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 04:57:08,566 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 04:57:08,566 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682657828566"}]},"ts":"9223372036854775807"} 2023-04-28 04:57:08,568 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 04:57:08,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 04:57:08,658 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,660 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 132 msec 2023-04-28 04:57:08,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 04:57:08,832 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-28 04:57:08,841 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 224) - Thread LEAK? -, OpenFileDescriptor=540 (was 539) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=476 (was 395) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=10265 (was 9958) - AvailableMemoryMB LEAK? - 2023-04-28 04:57:08,852 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=540, MaxFileDescriptor=60000, SystemLoadAverage=476, ProcessCount=171, AvailableMemoryMB=10264 2023-04-28 04:57:08,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:57:08,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$4(2112): Client=jenkins//136.243.104.168 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:57:08,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:08,857 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:57:08,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(697): Client=jenkins//136.243.104.168 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-28 04:57:08,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 04:57:08,859 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:57:08,860 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:08,861 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 empty. 2023-04-28 04:57:08,861 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:08,861 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:57:08,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 04:57:09,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 04:57:09,272 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:57:09,273 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 44c8ceb2d8e851788f33fa323eec3cf0, NAME => 'TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:57:09,284 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:57:09,284 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 44c8ceb2d8e851788f33fa323eec3cf0, disabling compactions & flushes 2023-04-28 04:57:09,284 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,284 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,284 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. after waiting 0 ms 2023-04-28 04:57:09,284 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,284 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,284 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:09,286 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:57:09,287 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657829287"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657829287"}]},"ts":"1682657829287"} 2023-04-28 04:57:09,288 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:57:09,288 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:57:09,289 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657829289"}]},"ts":"1682657829289"} 2023-04-28 04:57:09,289 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 04:57:09,348 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, ASSIGN}] 2023-04-28 04:57:09,349 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, ASSIGN 2023-04-28 04:57:09,350 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:57:09,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 04:57:09,501 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:09,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657829501"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657829501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657829501"}]},"ts":"1682657829501"} 2023-04-28 04:57:09,502 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:57:09,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44c8ceb2d8e851788f33fa323eec3cf0, NAME => 'TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:57:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:57:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,659 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,660 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A 2023-04-28 04:57:09,660 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A 2023-04-28 04:57:09,660 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:09,660 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8ceb2d8e851788f33fa323eec3cf0 columnFamilyName A 2023-04-28 04:57:09,660 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(310): Store=44c8ceb2d8e851788f33fa323eec3cf0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:09,661 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,662 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B 2023-04-28 04:57:09,662 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B 2023-04-28 04:57:09,662 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:09,662 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8ceb2d8e851788f33fa323eec3cf0 columnFamilyName B 2023-04-28 04:57:09,663 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(310): Store=44c8ceb2d8e851788f33fa323eec3cf0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:09,663 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,664 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C 2023-04-28 04:57:09,664 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C 2023-04-28 04:57:09,664 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:09,664 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8ceb2d8e851788f33fa323eec3cf0 columnFamilyName C 2023-04-28 04:57:09,665 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(310): Store=44c8ceb2d8e851788f33fa323eec3cf0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:09,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:57:09,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:09,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:57:09,672 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 44c8ceb2d8e851788f33fa323eec3cf0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64840720, jitterRate=-0.0337979793548584}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:57:09,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:09,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., pid=37, masterSystemTime=1682657829654 2023-04-28 04:57:09,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:09,674 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:09,675 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657829674"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657829674"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657829674"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657829674"}]},"ts":"1682657829674"} 2023-04-28 04:57:09,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-28 04:57:09,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 173 msec 2023-04-28 04:57:09,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-28 04:57:09,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, ASSIGN in 329 msec 2023-04-28 04:57:09,678 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:57:09,678 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657829678"}]},"ts":"1682657829678"} 2023-04-28 04:57:09,679 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 04:57:09,685 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:57:09,687 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 831 msec 2023-04-28 04:57:09,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 04:57:09,961 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-28 04:57:09,961 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x29738198 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:09,969 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1eaa86ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:09,970 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:09,972 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:09,973 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:57:09,974 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:53538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:57:09,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:57:09,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$12(2454): Client=jenkins//136.243.104.168 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:57:09,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:10,390 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 04:57:10,391 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:57:10,391 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 04:57:10,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 04:57:10,396 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, REOPEN/MOVE}] 2023-04-28 04:57:10,397 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, REOPEN/MOVE 2023-04-28 04:57:10,397 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:10,398 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657830397"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657830397"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657830397"}]},"ts":"1682657830397"} 2023-04-28 04:57:10,399 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:57:10,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 44c8ceb2d8e851788f33fa323eec3cf0, disabling compactions & flushes 2023-04-28 04:57:10,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. after waiting 0 ms 2023-04-28 04:57:10,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 04:57:10,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:10,562 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 44c8ceb2d8e851788f33fa323eec3cf0 to self. 2023-04-28 04:57:10,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,564 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=CLOSED 2023-04-28 04:57:10,564 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657830564"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657830564"}]},"ts":"1682657830564"} 2023-04-28 04:57:10,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-28 04:57:10,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 167 msec 2023-04-28 04:57:10,568 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=true 2023-04-28 04:57:10,719 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:10,719 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657830719"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657830719"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657830719"}]},"ts":"1682657830719"} 2023-04-28 04:57:10,720 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:57:10,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44c8ceb2d8e851788f33fa323eec3cf0, NAME => 'TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:57:10,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:57:10,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,878 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,879 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A 2023-04-28 04:57:10,879 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A 2023-04-28 04:57:10,879 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:10,879 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8ceb2d8e851788f33fa323eec3cf0 columnFamilyName A 2023-04-28 04:57:10,881 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(310): Store=44c8ceb2d8e851788f33fa323eec3cf0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:10,881 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,882 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B 2023-04-28 04:57:10,882 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B 2023-04-28 04:57:10,882 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:10,882 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8ceb2d8e851788f33fa323eec3cf0 columnFamilyName B 2023-04-28 04:57:10,883 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(310): Store=44c8ceb2d8e851788f33fa323eec3cf0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:10,883 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,884 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C 2023-04-28 04:57:10,884 DEBUG [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C 2023-04-28 04:57:10,884 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:10,884 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8ceb2d8e851788f33fa323eec3cf0 columnFamilyName C 2023-04-28 04:57:10,884 INFO [StoreOpener-44c8ceb2d8e851788f33fa323eec3cf0-1] regionserver.HStore(310): Store=44c8ceb2d8e851788f33fa323eec3cf0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:10,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:57:10,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:10,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 44c8ceb2d8e851788f33fa323eec3cf0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60605349, jitterRate=-0.09690992534160614}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:57:10,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:10,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., pid=42, masterSystemTime=1682657830872 2023-04-28 04:57:10,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:10,892 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:10,892 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657830892"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657830892"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657830892"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657830892"}]},"ts":"1682657830892"} 2023-04-28 04:57:10,894 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-28 04:57:10,894 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 173 msec 2023-04-28 04:57:10,895 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-28 04:57:10,895 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, REOPEN/MOVE in 498 msec 2023-04-28 04:57:10,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-28 04:57:10,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 503 msec 2023-04-28 04:57:10,898 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 922 msec 2023-04-28 04:57:10,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-28 04:57:10,899 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x28a98302 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:10,969 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d63a8ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:10,969 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x39d5c003 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:10,980 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d80e5a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:10,980 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x39e3a826 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:10,991 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ea1a2c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:10,991 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x019e7fff to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,001 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25185572, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,002 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x47e30d1c to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,011 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35f0f398, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,012 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x69fd9cc7 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,022 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ba3ce53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,022 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x54da4907 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,033 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5144fcff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,033 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x52cea5eb to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,043 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4cc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,043 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x3546dad6 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,053 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1280c068, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,054 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x4965c362 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:11,064 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4af630f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:11,073 DEBUG [hconnection-0x30b05cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,074 DEBUG [hconnection-0x2f03f8bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,073 DEBUG [hconnection-0x53c0b1a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,074 DEBUG [hconnection-0x16dfb289-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:11,075 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,078 DEBUG [hconnection-0x1ab7bca3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,078 DEBUG [hconnection-0x2e5d1ca7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,078 DEBUG [hconnection-0x15809007-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,078 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,078 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,080 DEBUG [hconnection-0x35e775a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,080 DEBUG [hconnection-0x6724eb6e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,080 DEBUG [hconnection-0x22c8721f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:11,082 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,082 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,084 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,085 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55850, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,088 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,089 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:11,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:11,092 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:11,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:11,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:11,094 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:55846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:11,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:11,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:57:11,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:11,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:11,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:11,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:11,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:11,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:11,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:11,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:11,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:11,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:11,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:11,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:11,111 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:11,112 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:11,112 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:11,112 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:11,112 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:11,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:11,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:11,112 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:11,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,126 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:11,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:11,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c869cd[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:11,128 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657831122, max timestamp=1682657831126Num uniques 859; 2023-04-28 04:57:11,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657831122, max timestamp=1682657831126Num uniques 859; 2023-04-28 04:57:11,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:11,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:11,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657831122, max timestamp=1682657831126Num uniques 859; 2023-04-28 04:57:11,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:11,137 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:11,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:11,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:11,137 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,138 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:11,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,147 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657891150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:11,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657891150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,151 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:11,151 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:11,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657891151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,152 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:11,152 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657891151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657891151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,158 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,158 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,158 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,162 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,165 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:11,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:11,178 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:11,178 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,178 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657891252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657891253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657891253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657891253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657891260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,278 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:11,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:11,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:11,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:11,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:11,292 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:11,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:11,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:11,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,292 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,304 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,304 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:11,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:11,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:11,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:11,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:11,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,308 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:11,312 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:11,312 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,312 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:11,312 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:11,312 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:11,312 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,312 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:11,312 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:11,312 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,315 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:11,315 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:11,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,326 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:11,328 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a35a818[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:11,328 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:11,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:11,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:11,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:11,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:11,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,348 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:11,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:11,350 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:11,350 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:11,350 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:11,350 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:11,350 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:11,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,359 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,370 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,370 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,370 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,370 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:11,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:11,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,370 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:11,371 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:11,371 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,371 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:11,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,371 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,371 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,371 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,371 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657891455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657891455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657891456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657891457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657891465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,471 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:11,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:11,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:11,475 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:11,475 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,475 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:11,475 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:11,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:11,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:11,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:11,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:11,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:11,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:11,496 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:11,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:11,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:11,497 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:11,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:11,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:11,497 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:11,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:11,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:11,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@577d6373[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:11,510 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:11,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:11,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:11,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:11,528 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042825568dc837dc426a8ed3bf684946ce6e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042825568dc837dc426a8ed3bf684946ce6e 2023-04-28 04:57:11,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,622 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:11,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:11,625 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:11,625 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:11,625 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:11,642 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,642 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,642 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,643 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:11,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:11,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,653 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:11,653 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:11,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,753 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:11,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:11,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:11,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:11,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:11,757 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:11,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:11,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:11,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657891759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657891760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657891761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,768 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657891761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,768 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:11,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:11,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:11,772 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:11,772 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:11,772 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:11,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:11,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657891771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,784 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:11,784 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,784 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:11,784 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:11,785 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:11,785 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:11,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:11,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:11,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:11,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:11,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:11,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:11,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7e3bfe[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:11,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:11,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,853 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:11,853 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:11,853 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:11,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:11,934 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/7061627572c245dbadd2a592eba2284a 2023-04-28 04:57:11,943 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:11,943 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,944 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:11,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:11,949 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:11,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:11,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:11,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:11,949 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:11,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/1d9f646f6318442dbdba108b894579d3 2023-04-28 04:57:11,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:11,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:11,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:11,981 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,990 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:11,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:11,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:11,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:11,990 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:11,990 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:11,990 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:11,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:12,090 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:12,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:12,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:12,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:12,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:12,094 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:12,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:12,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:12,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:12,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:12,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:12,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:12,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:12,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:12,108 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:12,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:12,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:12,108 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:12,109 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:12,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:12,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:12,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:12,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:12,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,116 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:12,116 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,116 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@283fb1dd[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:12,119 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:12,119 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:12,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:12,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:12,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:12,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:12,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:12,137 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,137 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:12,138 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:12,138 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:12,138 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:12,138 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:12,138 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:12,138 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:12,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:12,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,148 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:12,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:12,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:12,150 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:12,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:12,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,244 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:12,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:12,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:12,245 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:12,245 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:12,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:12,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657892265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:12,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657892267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:12,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657892272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:12,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657892274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:12,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657892282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,319 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:57:12,345 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:12,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:12,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:12,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:12,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:12,348 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:12,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:12,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:12,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:12,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:12,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:12,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:12,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:12,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:12,364 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:12,364 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:12,364 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:12,364 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:12,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:12,365 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,365 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:12,365 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:12,368 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,368 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:12,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:12,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:12,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f3db9aa[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:12,370 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:12,370 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/4dc6872c3fc04b42a1657a590c5bdcea 2023-04-28 04:57:12,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:12,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:12,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:12,380 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,380 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:12,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/7061627572c245dbadd2a592eba2284a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/7061627572c245dbadd2a592eba2284a 2023-04-28 04:57:12,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/7061627572c245dbadd2a592eba2284a, entries=150, sequenceid=15, filesize=25.1 K 2023-04-28 04:57:12,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/1d9f646f6318442dbdba108b894579d3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1d9f646f6318442dbdba108b894579d3 2023-04-28 04:57:12,389 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:12,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:12,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:12,390 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1d9f646f6318442dbdba108b894579d3, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 04:57:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:12,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:12,395 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:12,395 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:12,395 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:12,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/4dc6872c3fc04b42a1657a590c5bdcea as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4dc6872c3fc04b42a1657a590c5bdcea 2023-04-28 04:57:12,395 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:12,395 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:12,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4dc6872c3fc04b42a1657a590c5bdcea, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 04:57:12,401 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1296ms, sequenceid=15, compaction requested=false 2023-04-28 04:57:12,401 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 04:57:12,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:12,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:12,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:12,438 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:12,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:12,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:12,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:12,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,453 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:12,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:12,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:12,453 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:12,554 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:12,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:12,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:12,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:12,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:12,558 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:12,558 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:12,558 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:12,558 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:12,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,569 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:12,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:12,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,573 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:12,573 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:12,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:12,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:12,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:12,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:12,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:12,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:12,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:12,575 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:12,575 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:12,575 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:12,575 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-28 04:57:12,575 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:12,576 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:12,576 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:12,576 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:12,576 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:12,576 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:12,601 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cad2458334b34f8783fbe00c0d5db947 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cad2458334b34f8783fbe00c0d5db947 2023-04-28 04:57:13,006 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/18b9717069354628a224ade76cc6bf88 2023-04-28 04:57:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:13,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:13,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657893304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657893304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657893304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657893305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657893306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,415 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/b8a95ae03d524cfd8c446261238550af 2023-04-28 04:57:13,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657893415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657893415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657893415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657893415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657893416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657893618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657893618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657893619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657893620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657893621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,823 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/4671e81055ef4e13ad5cac6cd75f6004 2023-04-28 04:57:13,827 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/18b9717069354628a224ade76cc6bf88 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/18b9717069354628a224ade76cc6bf88 2023-04-28 04:57:13,831 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/18b9717069354628a224ade76cc6bf88, entries=150, sequenceid=50, filesize=25.1 K 2023-04-28 04:57:13,832 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/b8a95ae03d524cfd8c446261238550af as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b8a95ae03d524cfd8c446261238550af 2023-04-28 04:57:13,835 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b8a95ae03d524cfd8c446261238550af, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 04:57:13,836 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/4671e81055ef4e13ad5cac6cd75f6004 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4671e81055ef4e13ad5cac6cd75f6004 2023-04-28 04:57:13,839 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4671e81055ef4e13ad5cac6cd75f6004, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 04:57:13,841 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1266ms, sequenceid=50, compaction requested=false 2023-04-28 04:57:13,842 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:13,842 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:13,842 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:13,842 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:13,842 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,842 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:13,842 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:13,905 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:13,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:13,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:13,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:13,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:13,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:13,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:13,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63673fc5[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:13,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:13,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:13,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:13,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:13,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 04:57:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:13,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:13,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:13,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:13,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:13,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:13,931 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:13,931 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:13,931 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:13,931 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:13,931 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:13,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:13,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:13,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:13,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:13,939 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,944 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428783d83ec39794815b208c76bb15ddc0f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428783d83ec39794815b208c76bb15ddc0f 2023-04-28 04:57:13,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:13,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:13,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:13,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:13,949 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,949 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:13,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:13,949 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:13,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:13,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:13,949 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,950 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:13,950 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:13,950 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,950 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:13,950 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:13,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:13,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657893948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657893949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657893950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657893951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:13,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657893952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:13,963 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea 2023-04-28 04:57:14,050 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:14,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:14,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:14,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:14,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:14,054 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:14,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:14,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:14,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657894054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657894055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657894056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657894060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657894060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:14,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:14,077 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,077 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:14,077 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:14,077 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:14,077 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:14,084 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,084 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,085 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:14,086 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d58499c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:14,086 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:14,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:14,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:14,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:14,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:14,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:14,107 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:14,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:14,108 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:14,108 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:14,108 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:14,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:14,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:14,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,127 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:14,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,127 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:14,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:14,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,227 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:14,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:14,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:14,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:14,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:14,230 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:14,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:14,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:14,231 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,231 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,242 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:14,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:14,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:14,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:14,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:14,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:14,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:14,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:14,244 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:14,244 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:14,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:14,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:14,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:14,253 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:14,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:14,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28bcaeb3[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:14,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657894256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657894256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,263 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:14,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:14,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:14,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:14,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657894261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657894262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657894263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,274 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,274 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,274 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:14,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:14,275 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:14,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:14,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:14,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:14,275 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:14,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,285 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,286 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,320 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:14,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,320 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,321 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,320 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:14,321 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:14,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/9fd4272d44d54e55842d026161c5748e 2023-04-28 04:57:14,421 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:14,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:14,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:14,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:14,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:14,424 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:14,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:14,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:14,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,424 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,442 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:14,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:14,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:14,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:14,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:14,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:14,445 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:14,446 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,446 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:14,446 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:14,446 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:14,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:14,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:14,446 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:14,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:14,454 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:14,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a7eb2e7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:14,454 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:14,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:14,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:14,464 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:14,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:14,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:14,475 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:14,475 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:14,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:14,476 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:14,476 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:14,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,485 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,486 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:14,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:14,496 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:14,496 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:14,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:14,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657894561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657894561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657894569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657894570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:14,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657894571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,596 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:14,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:14,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:14,599 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:14,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:14,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:14,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,630 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,630 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,631 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,631 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:14,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:14,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:14,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:14,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:14,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:14,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:14,642 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:14,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403d3f7a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:14,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,653 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:14,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:14,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:14,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:14,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:14,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:14,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:14,665 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:14,665 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:14,665 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:14,665 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:14,665 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:14,674 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:14,674 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,674 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:14,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,676 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:14,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:14,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:14,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,684 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:14,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,685 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:14,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,685 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:14,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:14,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:14,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/f444002247dc45bf8ef5d4468eba3ebb 2023-04-28 04:57:14,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea 2023-04-28 04:57:14,785 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:14,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:14,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:14,788 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:14,788 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,788 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:14,788 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:14,788 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,788 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea, entries=150, sequenceid=62, filesize=25.1 K 2023-04-28 04:57:14,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/9fd4272d44d54e55842d026161c5748e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/9fd4272d44d54e55842d026161c5748e 2023-04-28 04:57:14,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/9fd4272d44d54e55842d026161c5748e, entries=150, sequenceid=62, filesize=11.6 K 2023-04-28 04:57:14,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/f444002247dc45bf8ef5d4468eba3ebb as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/f444002247dc45bf8ef5d4468eba3ebb 2023-04-28 04:57:14,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:14,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:14,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:14,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:14,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:14,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/f444002247dc45bf8ef5d4468eba3ebb, entries=150, sequenceid=62, filesize=11.6 K 2023-04-28 04:57:14,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 44c8ceb2d8e851788f33fa323eec3cf0 in 878ms, sequenceid=62, compaction requested=true 2023-04-28 04:57:14,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:14,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:14,803 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:14,803 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:14,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:14,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:14,804 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:14,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:14,804 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/A is initiating minor compaction (all files) 2023-04-28 04:57:14,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:14,804 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:14,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:14,804 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/B is initiating minor compaction (all files) 2023-04-28 04:57:14,804 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/A in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:14,804 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/7061627572c245dbadd2a592eba2284a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/18b9717069354628a224ade76cc6bf88, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=75.4 K 2023-04-28 04:57:14,805 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7061627572c245dbadd2a592eba2284a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682657831102 2023-04-28 04:57:14,805 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 18b9717069354628a224ade76cc6bf88, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657831147 2023-04-28 04:57:14,805 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ba1038f8ce8d4fb4ac6e55257a7ad2ea, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682657833286 2023-04-28 04:57:14,804 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/B in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:14,808 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1d9f646f6318442dbdba108b894579d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b8a95ae03d524cfd8c446261238550af, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/9fd4272d44d54e55842d026161c5748e] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=34.8 K 2023-04-28 04:57:14,809 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1d9f646f6318442dbdba108b894579d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682657831102 2023-04-28 04:57:14,810 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b8a95ae03d524cfd8c446261238550af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657831147 2023-04-28 04:57:14,811 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9fd4272d44d54e55842d026161c5748e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682657833286 2023-04-28 04:57:14,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:14,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:14,820 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,820 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:14,820 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:57:14,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:14,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:14,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:14,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:14,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:14,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:14,833 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#B#compaction#269 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:14,861 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#A#compaction#270 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:14,874 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/8140378584464cd09bc8a81b12193308 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8140378584464cd09bc8a81b12193308 2023-04-28 04:57:14,878 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/B of 44c8ceb2d8e851788f33fa323eec3cf0 into 8140378584464cd09bc8a81b12193308(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:14,878 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:14,878 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/B, priority=13, startTime=1682657834803; duration=0sec 2023-04-28 04:57:14,879 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:14,879 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:14,880 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:14,880 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/C is initiating minor compaction (all files) 2023-04-28 04:57:14,880 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/C in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:14,880 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4dc6872c3fc04b42a1657a590c5bdcea, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4671e81055ef4e13ad5cac6cd75f6004, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/f444002247dc45bf8ef5d4468eba3ebb] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=34.8 K 2023-04-28 04:57:14,881 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4dc6872c3fc04b42a1657a590c5bdcea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682657831102 2023-04-28 04:57:14,882 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4671e81055ef4e13ad5cac6cd75f6004, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657831147 2023-04-28 04:57:14,882 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting f444002247dc45bf8ef5d4468eba3ebb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682657833286 2023-04-28 04:57:14,884 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ddca5a8de09d4704b4a89aba08619e5a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ddca5a8de09d4704b4a89aba08619e5a 2023-04-28 04:57:14,897 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#C#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:15,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:15,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:15,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657895083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657895089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657895090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657895090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657895090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657895191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657895191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657895201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657895201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657895201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,314 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/745b7dd795154960a8f1fa4788c4d384 2023-04-28 04:57:15,329 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/b7dab2fd6eaa45c2a9dd26ccdaff18be 2023-04-28 04:57:15,329 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/663617b983e5435fa6c66824a8c791fe as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/663617b983e5435fa6c66824a8c791fe 2023-04-28 04:57:15,334 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/C of 44c8ceb2d8e851788f33fa323eec3cf0 into 663617b983e5435fa6c66824a8c791fe(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:15,334 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:15,334 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/C, priority=13, startTime=1682657834803; duration=0sec 2023-04-28 04:57:15,334 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657895395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657895395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657895406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657895406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657895406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657895700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657895701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657895710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657895712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:15,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657895712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,741 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/9f47bbb57d374a8e8756fe6b06a7a8d3 2023-04-28 04:57:15,746 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/745b7dd795154960a8f1fa4788c4d384 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/745b7dd795154960a8f1fa4788c4d384 2023-04-28 04:57:15,750 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/745b7dd795154960a8f1fa4788c4d384, entries=150, sequenceid=82, filesize=25.1 K 2023-04-28 04:57:15,751 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/b7dab2fd6eaa45c2a9dd26ccdaff18be as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b7dab2fd6eaa45c2a9dd26ccdaff18be 2023-04-28 04:57:15,755 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b7dab2fd6eaa45c2a9dd26ccdaff18be, entries=150, sequenceid=82, filesize=11.6 K 2023-04-28 04:57:15,756 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/9f47bbb57d374a8e8756fe6b06a7a8d3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/9f47bbb57d374a8e8756fe6b06a7a8d3 2023-04-28 04:57:15,759 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/9f47bbb57d374a8e8756fe6b06a7a8d3, entries=150, sequenceid=82, filesize=11.6 K 2023-04-28 04:57:15,760 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 44c8ceb2d8e851788f33fa323eec3cf0 in 940ms, sequenceid=82, compaction requested=false 2023-04-28 04:57:15,760 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:15,760 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:15,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:15,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:15,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,760 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:15,761 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:15,810 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:15,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:15,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:15,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:15,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f969a48[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:15,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,821 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:15,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:15,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:15,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:15,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:15,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:15,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:15,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:15,833 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:15,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:15,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:15,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:15,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:15,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:15,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:15,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:15,842 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:15,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,880 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:15,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:15,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:15,881 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:15,881 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:15,881 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:15,881 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:15,981 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:15,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:15,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:15,984 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:15,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:15,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:15,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:15,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:15,984 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:16,000 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:16,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:16,000 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,000 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,000 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:16,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:16,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:16,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:16,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:16,001 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:16,001 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:16,001 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:16,001 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:16,002 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:16,002 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:16,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:16,083 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/5b7b369125784987967236800399a9eb as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/5b7b369125784987967236800399a9eb 2023-04-28 04:57:16,086 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/A of 44c8ceb2d8e851788f33fa323eec3cf0 into 5b7b369125784987967236800399a9eb(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:57:16,086 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:16,086 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/A, priority=13, startTime=1682657834803; duration=1sec 2023-04-28 04:57:16,086 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:16,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:16,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657896229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657896230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657896232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657896232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657896236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657896338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657896338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657896338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657896339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657896339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,416 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281a22dcdfdcd14a73bf23e3d38bfe3e5b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281a22dcdfdcd14a73bf23e3d38bfe3e5b 2023-04-28 04:57:16,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657896543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657896543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657896544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657896544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657896544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,822 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/abccef5be6b241e8b4a06d962089753b 2023-04-28 04:57:16,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657896848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657896848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657896849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657896849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:16,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:16,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657896850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,266 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/5012a39202804457928a9473a44e8aca 2023-04-28 04:57:17,277 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/762abe80cc5b4dc18f2b431f2c3b8ab6 2023-04-28 04:57:17,281 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/abccef5be6b241e8b4a06d962089753b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/abccef5be6b241e8b4a06d962089753b 2023-04-28 04:57:17,284 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/abccef5be6b241e8b4a06d962089753b, entries=150, sequenceid=97, filesize=25.1 K 2023-04-28 04:57:17,285 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/5012a39202804457928a9473a44e8aca as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5012a39202804457928a9473a44e8aca 2023-04-28 04:57:17,314 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5012a39202804457928a9473a44e8aca, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 04:57:17,315 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/762abe80cc5b4dc18f2b431f2c3b8ab6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/762abe80cc5b4dc18f2b431f2c3b8ab6 2023-04-28 04:57:17,318 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/762abe80cc5b4dc18f2b431f2c3b8ab6, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 04:57:17,319 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1317ms, sequenceid=97, compaction requested=true 2023-04-28 04:57:17,319 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:17,319 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:17,319 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:17,319 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:17,319 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,319 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:17,319 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:17,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:17,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32926a3f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:17,349 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:17,349 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:17,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:57:17,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:17,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:17,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:17,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:17,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:17,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:17,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657897376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657897376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657897377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657897377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657897378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:17,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:17,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:17,473 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,473 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:17,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657897484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657897484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657897484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657897484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657897485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,529 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:17,530 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:17,530 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:17,530 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:17,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:17,532 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:17,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:17,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:17,532 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:17,532 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:17,546 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,546 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,546 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:17,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,550 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:17,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:17,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,558 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:17,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:17,559 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:17,559 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:17,659 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:17,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:17,662 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:17,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:17,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:17,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:17,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,673 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,674 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:17,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:17,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:17,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:17,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:17,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:17,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:17,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:17,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:17,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:17,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:17,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:17,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:17,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:17,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:17,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:17,676 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:17,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:17,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:17,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d941f00[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:17,687 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:17,687 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657897688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657897695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657897695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,700 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,700 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,700 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:17,700 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:17,700 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:17,700 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:17,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657897700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:17,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:17,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:17,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657897702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,708 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:17,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:17,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:17,708 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:17,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:17,708 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:17,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:17,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:17,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:17,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:17,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:17,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:17,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:17,727 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:17,727 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:17,776 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042833c4f3ff349c43f7a557ba3b631b807f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042833c4f3ff349c43f7a557ba3b631b807f 2023-04-28 04:57:17,789 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/893d5fb0ccb849318de98e794153a050 2023-04-28 04:57:17,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/83f2baba1add449abe2f1b709c25688f 2023-04-28 04:57:17,827 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:17,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:17,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:17,848 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:17,848 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:17,848 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:17,848 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:17,849 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,849 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:17,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:17,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:17,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:17,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:17,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:17,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:17,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:17,860 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:17,861 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:17,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:17,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:17,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:17,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:17,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:17,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:17,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:17,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:17,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:17,874 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:17,877 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:17,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747b3d27[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:17,877 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,884 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,884 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:17,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:17,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:17,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:17,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:17,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:17,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:17,896 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:17,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:17,899 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:17,899 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:17,899 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:17,899 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:17,899 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:17,905 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,905 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,906 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:17,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:17,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:17,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:17,909 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:17,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:17,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:17,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:17,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:17,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:17,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:17,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:17,917 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:17,918 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:17,918 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:17,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:18,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657897995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657898000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657898002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657898004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657898009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,018 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:18,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:18,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:18,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:18,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:18,021 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:18,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:18,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:18,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:18,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:18,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:18,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:18,033 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,033 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:18,033 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:18,033 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:18,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:18,072 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:18,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff4ad3a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:18,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:18,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:18,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:18,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:18,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:18,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:18,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:18,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:18,108 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:18,108 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:18,108 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:18,108 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:18,108 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:18,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:18,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:18,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:18,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,127 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:18,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,127 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:18,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,127 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,227 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:18,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:18,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:18,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:18,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:18,230 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:18,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:18,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:18,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,242 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:18,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:18,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:18,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:18,244 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,244 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:18,244 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:18,244 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:18,244 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:18,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,253 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:18,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:18,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e16eb69[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:18,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/2da2734b1bc846a09bd2f963341d9754 2023-04-28 04:57:18,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/893d5fb0ccb849318de98e794153a050 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/893d5fb0ccb849318de98e794153a050 2023-04-28 04:57:18,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/893d5fb0ccb849318de98e794153a050, entries=200, sequenceid=118, filesize=31.9 K 2023-04-28 04:57:18,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/83f2baba1add449abe2f1b709c25688f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/83f2baba1add449abe2f1b709c25688f 2023-04-28 04:57:18,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/83f2baba1add449abe2f1b709c25688f, entries=150, sequenceid=118, filesize=11.6 K 2023-04-28 04:57:18,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/2da2734b1bc846a09bd2f963341d9754 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/2da2734b1bc846a09bd2f963341d9754 2023-04-28 04:57:18,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/2da2734b1bc846a09bd2f963341d9754, entries=150, sequenceid=118, filesize=11.6 K 2023-04-28 04:57:18,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 44c8ceb2d8e851788f33fa323eec3cf0 in 932ms, sequenceid=118, compaction requested=true 2023-04-28 04:57:18,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:18,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:18,289 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:18,289 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:18,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:18,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:18,291 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:18,291 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/B is initiating minor compaction (all files) 2023-04-28 04:57:18,291 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/B in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,291 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8140378584464cd09bc8a81b12193308, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b7dab2fd6eaa45c2a9dd26ccdaff18be, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5012a39202804457928a9473a44e8aca, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/83f2baba1add449abe2f1b709c25688f] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=46.5 K 2023-04-28 04:57:18,291 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8140378584464cd09bc8a81b12193308, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682657833286 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/A is initiating minor compaction (all files) 2023-04-28 04:57:18,292 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/A in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,292 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/5b7b369125784987967236800399a9eb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/745b7dd795154960a8f1fa4788c4d384, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/abccef5be6b241e8b4a06d962089753b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/893d5fb0ccb849318de98e794153a050] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=107.4 K 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b7dab2fd6eaa45c2a9dd26ccdaff18be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682657833939 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5b7b369125784987967236800399a9eb, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682657833286 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5012a39202804457928a9473a44e8aca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682657835076 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 745b7dd795154960a8f1fa4788c4d384, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682657833939 2023-04-28 04:57:18,292 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 83f2baba1add449abe2f1b709c25688f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682657836224 2023-04-28 04:57:18,293 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting abccef5be6b241e8b4a06d962089753b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682657835076 2023-04-28 04:57:18,293 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 893d5fb0ccb849318de98e794153a050, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682657836224 2023-04-28 04:57:18,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:18,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:18,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:18,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:18,303 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#B#compaction#281 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:18,308 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#A#compaction#282 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:18,313 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/5bc962ca118b419c86ce1742ec991e71 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5bc962ca118b419c86ce1742ec991e71 2023-04-28 04:57:18,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:18,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,316 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/B of 44c8ceb2d8e851788f33fa323eec3cf0 into 5bc962ca118b419c86ce1742ec991e71(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:18,316 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,316 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/B, priority=12, startTime=1682657838289; duration=0sec 2023-04-28 04:57:18,317 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,317 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,318 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:18,318 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/C is initiating minor compaction (all files) 2023-04-28 04:57:18,318 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/C in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:18,318 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/663617b983e5435fa6c66824a8c791fe, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/9f47bbb57d374a8e8756fe6b06a7a8d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/762abe80cc5b4dc18f2b431f2c3b8ab6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/2da2734b1bc846a09bd2f963341d9754] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=46.5 K 2023-04-28 04:57:18,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:18,318 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:18,318 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:18,318 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:18,318 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:18,318 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:18,319 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 663617b983e5435fa6c66824a8c791fe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682657833286 2023-04-28 04:57:18,319 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9f47bbb57d374a8e8756fe6b06a7a8d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682657833939 2023-04-28 04:57:18,319 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 762abe80cc5b4dc18f2b431f2c3b8ab6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682657835076 2023-04-28 04:57:18,320 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 2da2734b1bc846a09bd2f963341d9754, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682657836224 2023-04-28 04:57:18,324 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#C#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:18,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,401 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:18,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:18,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:18,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,475 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,475 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,474 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,475 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,475 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,475 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:18,475 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:18,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:18,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:57:18,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:18,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:18,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:18,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:18,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:18,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:18,519 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428be4314cd85d9483b8d9ff471c21958f1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428be4314cd85d9483b8d9ff471c21958f1 2023-04-28 04:57:18,529 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/2fa3c2988cfe4f868c5a8475d874dd50 2023-04-28 04:57:18,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657898531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657898534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657898535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657898538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657898539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/d160edafda1f43debaf77629a0e65e05 2023-04-28 04:57:18,575 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:18,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:18,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:18,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:18,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:18,584 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:18,584 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,584 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:18,584 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:18,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,585 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:18,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:18,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:18,608 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:18,608 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,608 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:18,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:18,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:18,612 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,612 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:18,612 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:18,612 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:18,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:18,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:18,612 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:18,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:18,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:18,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,624 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63878bb4[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:18,625 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,625 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:18,631 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:18,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:18,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:18,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:18,642 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,643 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:18,643 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,643 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:18,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:18,644 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:18,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:18,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:18,644 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:18,645 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:18,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657898641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,652 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,652 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,652 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,653 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657898646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657898647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657898648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657898649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,655 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,720 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:18,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:18,721 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:18,721 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:18,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:18,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,732 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/3a15ccdaa33949379ab89f61ead38c66 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3a15ccdaa33949379ab89f61ead38c66 2023-04-28 04:57:18,741 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/A of 44c8ceb2d8e851788f33fa323eec3cf0 into 3a15ccdaa33949379ab89f61ead38c66(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:18,741 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:18,741 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/A, priority=12, startTime=1682657838289; duration=0sec 2023-04-28 04:57:18,741 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:18,742 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/e562379d3936459398a755f997057291 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e562379d3936459398a755f997057291 2023-04-28 04:57:18,750 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/C of 44c8ceb2d8e851788f33fa323eec3cf0 into e562379d3936459398a755f997057291(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:18,751 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:18,751 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/C, priority=12, startTime=1682657838289; duration=0sec 2023-04-28 04:57:18,751 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:18,821 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:18,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:18,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:18,824 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:18,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:18,824 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:18,825 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,825 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,842 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,842 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,842 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:18,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:18,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:18,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:18,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:18,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:18,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:18,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:18,843 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,843 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:18,843 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:18,843 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:18,843 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:18,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:18,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:18,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:18,852 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657898848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,853 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,853 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:18,856 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:18,856 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb810e6[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:18,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657898855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657898856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657898856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:18,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657898858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,889 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,889 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:18,889 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:18,889 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,889 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:18,889 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:18,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:18,905 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:18,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:18,907 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:18,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:18,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:18,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:18,907 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:18,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:18,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:18,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,919 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:18,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:18,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:18,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:18,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:18,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:18,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:18,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:18,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:18,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:18,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:18,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:18,927 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:18,927 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:18,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/1162e94e6bc14c04977f13b8678c4fd3 2023-04-28 04:57:18,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/2fa3c2988cfe4f868c5a8475d874dd50 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/2fa3c2988cfe4f868c5a8475d874dd50 2023-04-28 04:57:18,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/2fa3c2988cfe4f868c5a8475d874dd50, entries=200, sequenceid=132, filesize=32.0 K 2023-04-28 04:57:18,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/d160edafda1f43debaf77629a0e65e05 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d160edafda1f43debaf77629a0e65e05 2023-04-28 04:57:18,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d160edafda1f43debaf77629a0e65e05, entries=150, sequenceid=132, filesize=11.7 K 2023-04-28 04:57:18,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/1162e94e6bc14c04977f13b8678c4fd3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/1162e94e6bc14c04977f13b8678c4fd3 2023-04-28 04:57:18,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/1162e94e6bc14c04977f13b8678c4fd3, entries=150, sequenceid=132, filesize=11.7 K 2023-04-28 04:57:19,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 44c8ceb2d8e851788f33fa323eec3cf0 in 492ms, sequenceid=132, compaction requested=false 2023-04-28 04:57:19,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:19,027 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:19,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:19,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:19,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:19,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:19,031 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:19,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:19,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:19,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:19,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:19,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:19,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:19,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:19,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:19,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:19,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:19,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:19,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:19,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:19,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:19,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:19,047 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:19,047 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:19,047 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:19,047 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:19,048 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:19,086 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ad63888fb9cc4104b21a3da0ecb6bfa8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ad63888fb9cc4104b21a3da0ecb6bfa8 2023-04-28 04:57:19,096 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/8ad84c4701d045d8ae5129054d61c593 2023-04-28 04:57:19,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:19,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657899171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657899172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657899174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657899175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657899176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657899279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657899281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657899282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657899284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657899284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657899485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657899485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657899486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657899489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657899490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,523 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/8a889d0ea1804410a47e9e49a203ca6b 2023-04-28 04:57:19,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657899790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657899790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657899790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657899792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:19,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657899795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,933 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/0f197484eb544909b16c73cc052b52b8 2023-04-28 04:57:19,937 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/8ad84c4701d045d8ae5129054d61c593 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/8ad84c4701d045d8ae5129054d61c593 2023-04-28 04:57:19,940 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/8ad84c4701d045d8ae5129054d61c593, entries=150, sequenceid=153, filesize=25.3 K 2023-04-28 04:57:19,942 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/8a889d0ea1804410a47e9e49a203ca6b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8a889d0ea1804410a47e9e49a203ca6b 2023-04-28 04:57:19,958 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8a889d0ea1804410a47e9e49a203ca6b, entries=150, sequenceid=153, filesize=11.8 K 2023-04-28 04:57:19,959 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/0f197484eb544909b16c73cc052b52b8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0f197484eb544909b16c73cc052b52b8 2023-04-28 04:57:19,965 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0f197484eb544909b16c73cc052b52b8, entries=150, sequenceid=153, filesize=11.8 K 2023-04-28 04:57:19,966 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 44c8ceb2d8e851788f33fa323eec3cf0 in 919ms, sequenceid=153, compaction requested=true 2023-04-28 04:57:19,966 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:19,966 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:19,966 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:19,966 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:19,966 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:19,967 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:19,967 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:20,019 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:20,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:20,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:20,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:20,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543fe976[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:20,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:20,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:20,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:20,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:20,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:20,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,042 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:20,042 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:20,042 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:20,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:20,044 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:20,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:20,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:20,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:20,045 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:20,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:20,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:20,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:20,055 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:20,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:20,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,064 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:20,065 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:20,064 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:20,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:20,165 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:20,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:20,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:20,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:20,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:20,168 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:20,168 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:20,168 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:20,168 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:20,168 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,168 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:20,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:20,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:20,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:20,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:20,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:20,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:20,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:20,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:20,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:20,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:20,183 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:20,183 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:20,183 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:20,184 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:57:20,184 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:20,184 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:20,184 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:20,184 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:20,184 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:20,184 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:20,206 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c6e524a40fd34098841f02bd3d587297 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c6e524a40fd34098841f02bd3d587297 2023-04-28 04:57:20,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:20,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:20,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657900328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657900329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657900330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657900333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657900334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657900441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657900448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657900448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657900448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657900449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,627 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=166, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/67df4a7891e84382bb58cdf4cb5ca637 2023-04-28 04:57:20,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657900648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657900655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657900656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657900656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657900657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,664 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/08468d72aa1c4cb2b7ce342571a16585 2023-04-28 04:57:20,697 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/a20eec15d1c04b179b65cb7c89444da9 2023-04-28 04:57:20,710 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/67df4a7891e84382bb58cdf4cb5ca637 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/67df4a7891e84382bb58cdf4cb5ca637 2023-04-28 04:57:20,716 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/67df4a7891e84382bb58cdf4cb5ca637, entries=150, sequenceid=166, filesize=25.3 K 2023-04-28 04:57:20,719 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/08468d72aa1c4cb2b7ce342571a16585 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/08468d72aa1c4cb2b7ce342571a16585 2023-04-28 04:57:20,735 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/08468d72aa1c4cb2b7ce342571a16585, entries=150, sequenceid=166, filesize=11.8 K 2023-04-28 04:57:20,735 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/a20eec15d1c04b179b65cb7c89444da9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/a20eec15d1c04b179b65cb7c89444da9 2023-04-28 04:57:20,738 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/a20eec15d1c04b179b65cb7c89444da9, entries=150, sequenceid=166, filesize=11.8 K 2023-04-28 04:57:20,739 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 44c8ceb2d8e851788f33fa323eec3cf0 in 556ms, sequenceid=166, compaction requested=true 2023-04-28 04:57:20,739 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:20,739 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:20,739 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:20,739 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:20,739 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,739 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:20,739 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:20,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:20,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:20,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:20,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5b1d3b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:20,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:20,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:20,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:20,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:20,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:20,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:20,916 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:20,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:20,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:20,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:20,918 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:20,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:20,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:20,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:20,918 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:20,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:20,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:57:20,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:20,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:20,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:20,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:20,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:20,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:20,971 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,971 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,971 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:20,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:20,972 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,973 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657900977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657900989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657900988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657900988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:20,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657900989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:21,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:21,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:21,030 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:21,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:21,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:21,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,031 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:21,031 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:21,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:21,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657901089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657901096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657901096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657901096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657901096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,131 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:21,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:21,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:21,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:21,136 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:21,136 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:21,136 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:21,136 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:21,137 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,137 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,275 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:21,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:21,276 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,276 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:21,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:21,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:21,279 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:21,279 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:21,279 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:21,279 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:21,279 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:21,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657901300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657901302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657901302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657901302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657901303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,391 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042846a2c5845e5c460aafd3cc5e5a7fef3f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042846a2c5845e5c460aafd3cc5e5a7fef3f 2023-04-28 04:57:21,412 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,412 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:21,413 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,413 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22def30f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:21,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:21,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,468 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:21,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:21,468 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:21,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:21,603 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:21,603 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,604 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:21,604 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:21,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:21,606 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:21,606 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:21,606 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:21,606 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:21,606 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:21,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657901605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657901613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657901615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657901615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:21,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657901615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,646 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:21,646 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,646 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:21,647 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:21,647 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,762 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,762 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:21,762 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:21,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:21,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:21,762 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:21,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:21,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:21,762 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:21,763 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,763 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:21,763 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:21,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:21,795 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/9f3e06a0c0104a429405366812d040fa 2023-04-28 04:57:21,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/16645b748aa74471a653163eef994f09 2023-04-28 04:57:21,863 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:21,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:21,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:21,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:21,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:21,867 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:21,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:21,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:21,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:21,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,867 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,995 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:21,995 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,996 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,996 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:21,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:21,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:21,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:21,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:21,997 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:21,997 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:21,997 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:21,997 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:21,997 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:22,080 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,080 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:22,080 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,080 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ceff892[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:22,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:22,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657902110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657902120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657902121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657902121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657902122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/328b0e2eb1ef43a08f14ae40151b6fde 2023-04-28 04:57:22,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/9f3e06a0c0104a429405366812d040fa as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/9f3e06a0c0104a429405366812d040fa 2023-04-28 04:57:22,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/9f3e06a0c0104a429405366812d040fa, entries=200, sequenceid=186, filesize=32.1 K 2023-04-28 04:57:22,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/16645b748aa74471a653163eef994f09 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/16645b748aa74471a653163eef994f09 2023-04-28 04:57:22,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/16645b748aa74471a653163eef994f09, entries=150, sequenceid=186, filesize=11.8 K 2023-04-28 04:57:22,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/328b0e2eb1ef43a08f14ae40151b6fde as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/328b0e2eb1ef43a08f14ae40151b6fde 2023-04-28 04:57:22,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/328b0e2eb1ef43a08f14ae40151b6fde, entries=150, sequenceid=186, filesize=11.8 K 2023-04-28 04:57:22,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1270ms, sequenceid=186, compaction requested=true 2023-04-28 04:57:22,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:22,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:22,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:22,232 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:57:22,232 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:57:22,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:22,234 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143450 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:57:22,234 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60170 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:57:22,234 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/A is initiating minor compaction (all files) 2023-04-28 04:57:22,234 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/B is initiating minor compaction (all files) 2023-04-28 04:57:22,234 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/A in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:22,235 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/B in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:22,235 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5bc962ca118b419c86ce1742ec991e71, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d160edafda1f43debaf77629a0e65e05, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8a889d0ea1804410a47e9e49a203ca6b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/08468d72aa1c4cb2b7ce342571a16585, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/16645b748aa74471a653163eef994f09] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=58.8 K 2023-04-28 04:57:22,235 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3a15ccdaa33949379ab89f61ead38c66, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/2fa3c2988cfe4f868c5a8475d874dd50, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/8ad84c4701d045d8ae5129054d61c593, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/67df4a7891e84382bb58cdf4cb5ca637, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/9f3e06a0c0104a429405366812d040fa] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=140.1 K 2023-04-28 04:57:22,235 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5bc962ca118b419c86ce1742ec991e71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682657836224 2023-04-28 04:57:22,235 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3a15ccdaa33949379ab89f61ead38c66, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682657836224 2023-04-28 04:57:22,235 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d160edafda1f43debaf77629a0e65e05, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682657837375 2023-04-28 04:57:22,235 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 2fa3c2988cfe4f868c5a8475d874dd50, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682657837375 2023-04-28 04:57:22,235 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8a889d0ea1804410a47e9e49a203ca6b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682657838530 2023-04-28 04:57:22,235 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 8ad84c4701d045d8ae5129054d61c593, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682657838530 2023-04-28 04:57:22,236 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 08468d72aa1c4cb2b7ce342571a16585, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682657839171 2023-04-28 04:57:22,236 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 67df4a7891e84382bb58cdf4cb5ca637, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682657839171 2023-04-28 04:57:22,236 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 16645b748aa74471a653163eef994f09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682657840332 2023-04-28 04:57:22,238 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9f3e06a0c0104a429405366812d040fa, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682657840329 2023-04-28 04:57:22,247 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#B#compaction#297 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:22,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,252 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#A#compaction#296 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:22,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:22,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:22,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:22,253 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,253 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:22,258 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/a5c6da46257f4308b4f95be138194239 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/a5c6da46257f4308b4f95be138194239 2023-04-28 04:57:22,263 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/B of 44c8ceb2d8e851788f33fa323eec3cf0 into a5c6da46257f4308b4f95be138194239(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:22,263 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:22,263 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/B, priority=11, startTime=1682657842232; duration=0sec 2023-04-28 04:57:22,263 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:22,263 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:57:22,271 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60170 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:57:22,271 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/C is initiating minor compaction (all files) 2023-04-28 04:57:22,271 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/C in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:22,271 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e562379d3936459398a755f997057291, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/1162e94e6bc14c04977f13b8678c4fd3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0f197484eb544909b16c73cc052b52b8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/a20eec15d1c04b179b65cb7c89444da9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/328b0e2eb1ef43a08f14ae40151b6fde] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=58.8 K 2023-04-28 04:57:22,271 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting e562379d3936459398a755f997057291, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682657836224 2023-04-28 04:57:22,272 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1162e94e6bc14c04977f13b8678c4fd3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682657837375 2023-04-28 04:57:22,272 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 0f197484eb544909b16c73cc052b52b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682657838530 2023-04-28 04:57:22,272 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a20eec15d1c04b179b65cb7c89444da9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682657839171 2023-04-28 04:57:22,272 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 328b0e2eb1ef43a08f14ae40151b6fde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682657840332 2023-04-28 04:57:22,284 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:22,284 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:22,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,284 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:22,287 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#C#compaction#298 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:22,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:22,288 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:22,288 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:22,288 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:22,288 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:22,288 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:22,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:22,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:22,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:22,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,354 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:22,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:22,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:22,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:22,408 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:22,408 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:22,408 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,409 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:22,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:22,509 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:22,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:22,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:22,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:22,511 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:22,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:22,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:22,511 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:22,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,512 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:22,579 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:22,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:22,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:22,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:22,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:22,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:22,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:22,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:22,580 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:22,580 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:22,580 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:22,580 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:22,580 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:22,580 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:22,581 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:22,581 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:22,701 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/7ffa602b63414ae784df96d6a97d30c4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7ffa602b63414ae784df96d6a97d30c4 2023-04-28 04:57:22,704 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/C of 44c8ceb2d8e851788f33fa323eec3cf0 into 7ffa602b63414ae784df96d6a97d30c4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:22,704 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:22,704 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/C, priority=11, startTime=1682657842232; duration=0sec 2023-04-28 04:57:22,704 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:22,999 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289f793e98e0894f6daea2fafd3dea8207 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289f793e98e0894f6daea2fafd3dea8207 2023-04-28 04:57:23,003 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/a826b14c21c24271806093657210aa9b 2023-04-28 04:57:23,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:23,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:23,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657903148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657903148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657903152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657903153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657903154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657903255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657903255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657903256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657903260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657903260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,426 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/28ccade5d4fe4d93847a67fa56099879 2023-04-28 04:57:23,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657903460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657903461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657903462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657903465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657903466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,490 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/4d17c4f8ccc94287b0bd817397ee6a63 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/4d17c4f8ccc94287b0bd817397ee6a63 2023-04-28 04:57:23,493 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/A of 44c8ceb2d8e851788f33fa323eec3cf0 into 4d17c4f8ccc94287b0bd817397ee6a63(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:57:23,493 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:23,493 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/A, priority=11, startTime=1682657842232; duration=1sec 2023-04-28 04:57:23,493 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:23,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55784 deadline: 1682657903764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55810 deadline: 1682657903766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657903768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657903773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:23,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657903775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,835 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/7095953fd8154630b45449496f14f2c8 2023-04-28 04:57:23,839 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/a826b14c21c24271806093657210aa9b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/a826b14c21c24271806093657210aa9b 2023-04-28 04:57:23,842 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/a826b14c21c24271806093657210aa9b, entries=150, sequenceid=201, filesize=25.3 K 2023-04-28 04:57:23,843 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/28ccade5d4fe4d93847a67fa56099879 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/28ccade5d4fe4d93847a67fa56099879 2023-04-28 04:57:23,857 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/28ccade5d4fe4d93847a67fa56099879, entries=150, sequenceid=201, filesize=11.8 K 2023-04-28 04:57:23,861 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/7095953fd8154630b45449496f14f2c8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7095953fd8154630b45449496f14f2c8 2023-04-28 04:57:23,875 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7095953fd8154630b45449496f14f2c8, entries=150, sequenceid=201, filesize=11.8 K 2023-04-28 04:57:23,876 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1295ms, sequenceid=201, compaction requested=false 2023-04-28 04:57:23,876 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:23,876 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:23,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:23,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:23,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:23,876 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:23,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:23,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:23,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:23,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:23,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:23,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:23,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b0c020[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:23,928 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:23,928 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,936 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:23,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:23,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:23,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:23,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:23,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:23,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:23,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:23,949 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:23,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:23,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:23,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:23,949 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:23,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:23,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:23,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:23,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,959 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:23,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:23,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:23,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:23,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:23,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:23,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:23,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:23,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,968 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:23,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:23,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:23,968 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:23,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:23,969 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:24,069 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:24,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:24,073 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:24,073 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:24,073 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:24,073 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:24,073 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:24,073 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:24,084 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:24,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:24,084 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:24,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:24,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:24,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:24,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:24,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:24,086 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:24,086 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:24,086 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:24,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:24,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:24,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:24,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:24,096 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 04:57:24,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:24,097 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:24,097 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:24,097 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:24,097 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:24,097 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:24,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:24,273 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:24,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657904285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657904286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657904287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657904287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657904287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657904388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657904392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657904392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657904392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55814 deadline: 1682657904393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,517 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cce48a8b96524361aa29c74bfcc2aee0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cce48a8b96524361aa29c74bfcc2aee0 2023-04-28 04:57:24,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657904597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657904598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657904599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657904599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55850 deadline: 1682657904599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657904906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657904906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657904906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657904907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657904907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:24,951 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/6dbd522618f04c5986baab8d3233f040 2023-04-28 04:57:25,359 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/47d37e77808445679f536f37264b98dc 2023-04-28 04:57:25,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:25,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:55846 deadline: 1682657905415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:25,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657905419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:25,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657905419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:25,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657905420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:25,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657905420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,774 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/da9bc4c36ced47fc8eb040b3e2f4340b 2023-04-28 04:57:25,777 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/6dbd522618f04c5986baab8d3233f040 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6dbd522618f04c5986baab8d3233f040 2023-04-28 04:57:25,780 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6dbd522618f04c5986baab8d3233f040, entries=150, sequenceid=221, filesize=25.3 K 2023-04-28 04:57:25,783 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/47d37e77808445679f536f37264b98dc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47d37e77808445679f536f37264b98dc 2023-04-28 04:57:25,788 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47d37e77808445679f536f37264b98dc, entries=150, sequenceid=221, filesize=11.8 K 2023-04-28 04:57:25,789 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/da9bc4c36ced47fc8eb040b3e2f4340b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9bc4c36ced47fc8eb040b3e2f4340b 2023-04-28 04:57:25,793 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9bc4c36ced47fc8eb040b3e2f4340b, entries=150, sequenceid=221, filesize=11.8 K 2023-04-28 04:57:25,794 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1698ms, sequenceid=221, compaction requested=true 2023-04-28 04:57:25,794 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:25,794 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:25,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:25,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:25,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:25,794 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:25,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,800 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,800 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:25,803 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:25,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:25,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2227a728[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:25,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:25,805 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,810 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:25,810 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:25,810 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:25,810 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,810 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:25,820 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:25,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:25,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:25,823 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:25,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:25,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:25,823 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:25,823 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:25,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:25,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:25,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:25,832 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:25,833 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,871 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,871 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:25,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:25,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:25,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:25,871 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:25,871 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:25,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:25,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,872 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:25,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:25,872 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:25,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:25,972 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:25,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:25,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:25,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:25,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:25,974 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:25,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:25,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:25,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:25,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,975 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:25,989 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:25,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:25,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:25,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:25,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:25,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:25,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:25,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:25,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:25,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:25,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:25,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:25,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:25,990 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:25,990 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:25,991 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 04:57:25,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:25,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:25,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:25,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:25,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:25,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:26,408 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ea26e872de1a4db7936b83eaa9cb86ab to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ea26e872de1a4db7936b83eaa9cb86ab 2023-04-28 04:57:26,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:26,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:26,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657906445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657906446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657906446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657906447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657906454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657906555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657906556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657906556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657906556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657906563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657906759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657906766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657906766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657906767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:26,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657906770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:26,815 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=236, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/3c7334659dc24c93b9cca2763455b1ec 2023-04-28 04:57:27,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657907068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657907073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657907074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657907075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657907094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,224 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/d12d348cd79f436db5a6b6edc8476250 2023-04-28 04:57:27,247 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/0c69fc0ec3984c3cbe46ae1593219aef 2023-04-28 04:57:27,261 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/3c7334659dc24c93b9cca2763455b1ec as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3c7334659dc24c93b9cca2763455b1ec 2023-04-28 04:57:27,269 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3c7334659dc24c93b9cca2763455b1ec, entries=150, sequenceid=236, filesize=25.3 K 2023-04-28 04:57:27,270 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/d12d348cd79f436db5a6b6edc8476250 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d12d348cd79f436db5a6b6edc8476250 2023-04-28 04:57:27,278 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d12d348cd79f436db5a6b6edc8476250, entries=150, sequenceid=236, filesize=11.8 K 2023-04-28 04:57:27,279 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/0c69fc0ec3984c3cbe46ae1593219aef as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0c69fc0ec3984c3cbe46ae1593219aef 2023-04-28 04:57:27,289 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0c69fc0ec3984c3cbe46ae1593219aef, entries=150, sequenceid=236, filesize=11.8 K 2023-04-28 04:57:27,290 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1299ms, sequenceid=236, compaction requested=true 2023-04-28 04:57:27,290 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:27,290 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:27,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:27,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:27,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:27,290 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:27,374 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,374 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:27,375 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,375 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:27,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:27,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77347ee8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:27,376 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:27,376 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:27,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:27,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:27,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:27,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:27,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:27,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:27,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:27,508 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:27,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:27,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:27,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:27,508 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:27,515 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:27,515 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,515 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:27,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,516 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,516 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:27,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 04:57:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:27,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:27,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:27,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:27,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:27,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:27,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:27,590 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:27,590 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:27,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:27,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657907599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657907601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657907603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657907604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657907605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,690 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:27,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:27,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:27,695 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:27,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:27,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:27,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:27,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:27,705 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:27,705 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,705 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:27,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:27,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:27,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:27,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:27,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:27,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:27,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:27,706 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:27,707 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:27,707 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:27,707 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:27,707 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:27,707 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:27,707 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,707 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:27,707 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:27,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657907707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:27,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:27,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:27,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24cc23f4[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:27,717 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:27,717 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657907713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657907713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657907715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:27,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:27,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:27,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:27,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:27,736 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:27,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:27,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:27,738 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:27,738 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:27,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:27,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:27,739 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:27,826 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,827 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:27,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,828 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:27,828 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,829 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:27,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:27,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:27,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:27,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:27,862 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:27,862 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:27,861 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:27,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:27,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657907913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657907921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657907921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657907922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,962 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:27,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:27,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:27,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:27,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:27,967 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:27,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:27,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:27,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:27,968 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,968 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:27,978 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:27,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:27,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:27,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:27,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:27,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:27,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:27,980 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:27,981 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:27,981 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:27,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:27,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:27,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:27,991 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:27,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:27,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:27,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:27,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:27,999 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:27,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:27,999 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304284bc13fb7280340a5bfe1bed6b6355b25 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284bc13fb7280340a5bfe1bed6b6355b25 2023-04-28 04:57:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:28,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257c568d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:28,002 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:28,002 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,006 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/0c5570e0e4de437e9335a75faa4be5ff 2023-04-28 04:57:28,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:28,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:28,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:28,011 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,011 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:28,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:28,020 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:28,024 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:28,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:28,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:28,024 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:28,024 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:28,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,034 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:28,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:28,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,096 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:28,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,096 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:28,096 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,196 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:28,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:28,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:28,200 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:28,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:28,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:28,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,220 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,221 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,221 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:28,224 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:28,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:28,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:28,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:28,225 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:28,225 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,225 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:28,225 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:28,225 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:28,225 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,225 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:28,225 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:28,225 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,226 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:28,226 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:28,226 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,226 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:28,226 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:28,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657908220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657908226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657908227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657908227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,231 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,231 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:28,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3afdabed[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:28,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:28,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,242 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:28,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:28,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:28,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:28,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:28,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:28,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:28,260 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:28,260 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:28,260 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:28,260 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:28,260 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:28,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,276 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,284 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:28,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,284 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:28,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,284 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:28,384 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:28,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:28,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:28,388 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:28,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:28,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:28,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:28,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:28,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:28,401 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,401 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:28,401 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:28,401 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:28,401 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:28,410 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,410 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,410 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,410 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:28,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e6a4164[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:28,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:28,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/3c1cd980fb5942e8aa9c64968ee09ad7 2023-04-28 04:57:28,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,421 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:28,421 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:28,421 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:28,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:28,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:28,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,431 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:28,433 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:28,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:28,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:28,433 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:28,433 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:28,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,442 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,452 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:28,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,452 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:28,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,553 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:28,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:28,556 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:28,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:28,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:28,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:28,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:28,569 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,569 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:28,569 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:28,569 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:28,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:28,578 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:28,580 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:28,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfc0cde[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:28,580 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:28,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:28,589 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,589 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:28,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:28,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:28,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:28,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:28,601 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:28,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:28,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:28,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:28,601 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:28,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,610 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657908609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:28,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:28,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:28,621 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,621 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:28,621 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:28,721 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:28,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:28,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:28,725 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:28,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:28,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:28,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657908729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657908731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657908733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657908734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,791 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:28,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:28,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:28,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:28,795 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,795 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:28,795 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:28,795 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:28,795 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:28,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:28,796 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:28,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:28,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cf3ef12[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:28,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:28,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,821 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:28,822 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,822 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:28,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:28,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:28,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/87c074fdabf44e55884bef2a4701066f 2023-04-28 04:57:28,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/0c5570e0e4de437e9335a75faa4be5ff as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/0c5570e0e4de437e9335a75faa4be5ff 2023-04-28 04:57:28,832 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:28,832 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,832 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:28,832 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/0c5570e0e4de437e9335a75faa4be5ff, entries=200, sequenceid=254, filesize=32.1 K 2023-04-28 04:57:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/3c1cd980fb5942e8aa9c64968ee09ad7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/3c1cd980fb5942e8aa9c64968ee09ad7 2023-04-28 04:57:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:28,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:28,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:28,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:28,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:28,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:28,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:28,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/3c1cd980fb5942e8aa9c64968ee09ad7, entries=150, sequenceid=254, filesize=11.8 K 2023-04-28 04:57:28,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/87c074fdabf44e55884bef2a4701066f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/87c074fdabf44e55884bef2a4701066f 2023-04-28 04:57:28,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/87c074fdabf44e55884bef2a4701066f, entries=150, sequenceid=254, filesize=11.8 K 2023-04-28 04:57:28,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1259ms, sequenceid=254, compaction requested=true 2023-04-28 04:57:28,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:28,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:28,841 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:57:28,841 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:57:28,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:28,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:28,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,843 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:28,843 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60590 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:57:28,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,843 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/B is initiating minor compaction (all files) 2023-04-28 04:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,843 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/B in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,843 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136880 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:57:28,843 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/a5c6da46257f4308b4f95be138194239, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/28ccade5d4fe4d93847a67fa56099879, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47d37e77808445679f536f37264b98dc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d12d348cd79f436db5a6b6edc8476250, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/3c1cd980fb5942e8aa9c64968ee09ad7] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=59.2 K 2023-04-28 04:57:28,843 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/A is initiating minor compaction (all files) 2023-04-28 04:57:28,843 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/A in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,843 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/4d17c4f8ccc94287b0bd817397ee6a63, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/a826b14c21c24271806093657210aa9b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6dbd522618f04c5986baab8d3233f040, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3c7334659dc24c93b9cca2763455b1ec, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/0c5570e0e4de437e9335a75faa4be5ff] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=133.7 K 2023-04-28 04:57:28,844 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a5c6da46257f4308b4f95be138194239, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682657840332 2023-04-28 04:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,844 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4d17c4f8ccc94287b0bd817397ee6a63, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682657840332 2023-04-28 04:57:28,844 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 28ccade5d4fe4d93847a67fa56099879, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682657840987 2023-04-28 04:57:28,844 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting a826b14c21c24271806093657210aa9b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682657840987 2023-04-28 04:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,844 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 47d37e77808445679f536f37264b98dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682657843146 2023-04-28 04:57:28,844 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:28,844 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6dbd522618f04c5986baab8d3233f040, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682657843146 2023-04-28 04:57:28,845 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d12d348cd79f436db5a6b6edc8476250, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1682657844284 2023-04-28 04:57:28,845 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3c7334659dc24c93b9cca2763455b1ec, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1682657844284 2023-04-28 04:57:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,845 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 3c1cd980fb5942e8aa9c64968ee09ad7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682657846445 2023-04-28 04:57:28,845 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0c5570e0e4de437e9335a75faa4be5ff, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682657846445 2023-04-28 04:57:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:28,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:28,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:28,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:28,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:28,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:28,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,854 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,854 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,854 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:28,858 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#B#compaction#312 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:28,866 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#A#compaction#311 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:28,954 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:28,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:28,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:28,957 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:28,957 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,957 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:28,957 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:28,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:28,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:28,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:28,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:28,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:28,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:28,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:28,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:28,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:28,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:28,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:28,970 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:28,971 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:28,971 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:29,286 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/cb8b8cfa4a9640cda1a8122f1812ec4b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/cb8b8cfa4a9640cda1a8122f1812ec4b 2023-04-28 04:57:29,292 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/B of 44c8ceb2d8e851788f33fa323eec3cf0 into cb8b8cfa4a9640cda1a8122f1812ec4b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:29,292 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:29,293 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/B, priority=11, startTime=1682657848841; duration=0sec 2023-04-28 04:57:29,293 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:29,293 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:57:29,294 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60590 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:57:29,294 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/C is initiating minor compaction (all files) 2023-04-28 04:57:29,294 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/C in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:29,294 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7ffa602b63414ae784df96d6a97d30c4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7095953fd8154630b45449496f14f2c8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9bc4c36ced47fc8eb040b3e2f4340b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0c69fc0ec3984c3cbe46ae1593219aef, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/87c074fdabf44e55884bef2a4701066f] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=59.2 K 2023-04-28 04:57:29,295 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7ffa602b63414ae784df96d6a97d30c4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682657840332 2023-04-28 04:57:29,295 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7095953fd8154630b45449496f14f2c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682657840987 2023-04-28 04:57:29,295 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting da9bc4c36ced47fc8eb040b3e2f4340b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682657843146 2023-04-28 04:57:29,295 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 0c69fc0ec3984c3cbe46ae1593219aef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1682657844284 2023-04-28 04:57:29,296 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 87c074fdabf44e55884bef2a4701066f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682657846445 2023-04-28 04:57:29,309 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#C#compaction#314 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:29,384 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a49dd711b6ff496e9b9f0a24a82c53ad to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a49dd711b6ff496e9b9f0a24a82c53ad 2023-04-28 04:57:29,399 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/90cb022f4b644f23a6e36c76ac5baf15 2023-04-28 04:57:29,412 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/5e715c5ff5d24beca6a7df4309cd434b 2023-04-28 04:57:29,720 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/b609607cbecb4fe190839064cf518923 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/b609607cbecb4fe190839064cf518923 2023-04-28 04:57:29,724 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/C of 44c8ceb2d8e851788f33fa323eec3cf0 into b609607cbecb4fe190839064cf518923(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:29,724 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:29,724 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/C, priority=11, startTime=1682657848841; duration=0sec 2023-04-28 04:57:29,724 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:29,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:29,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:29,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657909767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657909767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657909768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657909768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,833 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/c54d4a759e70469e9ff1efa04980bede 2023-04-28 04:57:29,836 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/90cb022f4b644f23a6e36c76ac5baf15 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/90cb022f4b644f23a6e36c76ac5baf15 2023-04-28 04:57:29,839 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/90cb022f4b644f23a6e36c76ac5baf15, entries=150, sequenceid=268, filesize=25.4 K 2023-04-28 04:57:29,840 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/5e715c5ff5d24beca6a7df4309cd434b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5e715c5ff5d24beca6a7df4309cd434b 2023-04-28 04:57:29,843 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5e715c5ff5d24beca6a7df4309cd434b, entries=150, sequenceid=268, filesize=11.9 K 2023-04-28 04:57:29,844 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/c54d4a759e70469e9ff1efa04980bede as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/c54d4a759e70469e9ff1efa04980bede 2023-04-28 04:57:29,848 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/c54d4a759e70469e9ff1efa04980bede, entries=150, sequenceid=268, filesize=11.9 K 2023-04-28 04:57:29,848 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 44c8ceb2d8e851788f33fa323eec3cf0 in 877ms, sequenceid=268, compaction requested=false 2023-04-28 04:57:29,848 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:29,848 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:29,848 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:29,848 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:29,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:29,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:29,869 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,869 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:29,870 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:29,870 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:29,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:29,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b0683a2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:29,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:29,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:29,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 04:57:29,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:29,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:29,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:29,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:29,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:29,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:29,883 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,883 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:29,883 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:29,883 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:29,883 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,883 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:29,892 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a6190319f9e0410e80ee10f7fd1b3bf2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a6190319f9e0410e80ee10f7fd1b3bf2 2023-04-28 04:57:29,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:29,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:29,895 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:29,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657909892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657909897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:29,902 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:29,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:29,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:29,902 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:29,902 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:29,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657909898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,908 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/d230ff5bbc5a47ab8b78812d9e89665f 2023-04-28 04:57:29,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:29,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657909901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:29,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:29,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,924 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:29,924 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:29,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:29,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/923de6031e6c4b94afba1f8fea8b5c3f 2023-04-28 04:57:29,927 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:29,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:29,928 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:29,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:29,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:29,928 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:29,928 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:29,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:29,928 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:29,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:30,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657910001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657910002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657910006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657910011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,028 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:30,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:30,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:30,031 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:30,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:30,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:30,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:30,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:30,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:30,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,043 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:30,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:30,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:30,044 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,044 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:30,044 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:30,044 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:30,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:30,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,053 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:30,053 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,053 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:30,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db84c7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:30,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:30,055 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:30,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:30,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:30,064 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,064 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:30,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:30,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:30,074 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:30,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:30,076 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:30,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:30,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:30,076 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:30,076 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:30,085 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:30,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:30,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,086 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:30,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:30,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:30,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:30,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:30,096 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:30,096 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:30,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,097 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,111 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/6867156731ac4d499ee109b3bb1c0200 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6867156731ac4d499ee109b3bb1c0200 2023-04-28 04:57:30,115 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/A of 44c8ceb2d8e851788f33fa323eec3cf0 into 6867156731ac4d499ee109b3bb1c0200(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:57:30,115 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:30,115 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/A, priority=11, startTime=1682657848841; duration=1sec 2023-04-28 04:57:30,115 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:30,196 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:30,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:30,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:30,198 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:30,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:30,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:30,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:30,199 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,199 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:30,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:30,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:30,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:30,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:30,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:30,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:30,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:30,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:30,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:30,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:30,213 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:30,213 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,213 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:30,213 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:30,213 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,213 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:30,213 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:30,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657910210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657910210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657910211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657910217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,268 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:30,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,269 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:30,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:30,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c0af21e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:30,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:30,270 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,285 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,285 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:30,285 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,285 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:30,285 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:30,285 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:30,295 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:30,295 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:30,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,295 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:30,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:30,297 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:30,297 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:30,297 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:30,297 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:30,297 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:30,306 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,306 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:30,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,306 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:30,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,307 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:30,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:30,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:30,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:30,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:30,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:30,316 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:30,317 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:30,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,317 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:30,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:30,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:30,317 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/3fffbc83303641f58eef2e7bad5ae955 2023-04-28 04:57:30,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/d230ff5bbc5a47ab8b78812d9e89665f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d230ff5bbc5a47ab8b78812d9e89665f 2023-04-28 04:57:30,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d230ff5bbc5a47ab8b78812d9e89665f, entries=200, sequenceid=289, filesize=32.3 K 2023-04-28 04:57:30,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/923de6031e6c4b94afba1f8fea8b5c3f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/923de6031e6c4b94afba1f8fea8b5c3f 2023-04-28 04:57:30,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/923de6031e6c4b94afba1f8fea8b5c3f, entries=150, sequenceid=289, filesize=11.9 K 2023-04-28 04:57:30,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/3fffbc83303641f58eef2e7bad5ae955 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/3fffbc83303641f58eef2e7bad5ae955 2023-04-28 04:57:30,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/3fffbc83303641f58eef2e7bad5ae955, entries=150, sequenceid=289, filesize=11.9 K 2023-04-28 04:57:30,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 44c8ceb2d8e851788f33fa323eec3cf0 in 496ms, sequenceid=289, compaction requested=true 2023-04-28 04:57:30,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:30,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:30,374 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:30,374 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:30,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:30,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:30,375 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85608 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:30,375 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:30,375 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/A is initiating minor compaction (all files) 2023-04-28 04:57:30,375 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/B is initiating minor compaction (all files) 2023-04-28 04:57:30,375 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/A in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,375 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/B in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,375 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6867156731ac4d499ee109b3bb1c0200, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/90cb022f4b644f23a6e36c76ac5baf15, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d230ff5bbc5a47ab8b78812d9e89665f] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=83.6 K 2023-04-28 04:57:30,375 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/cb8b8cfa4a9640cda1a8122f1812ec4b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5e715c5ff5d24beca6a7df4309cd434b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/923de6031e6c4b94afba1f8fea8b5c3f] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=36.1 K 2023-04-28 04:57:30,376 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6867156731ac4d499ee109b3bb1c0200, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682657846445 2023-04-28 04:57:30,376 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting cb8b8cfa4a9640cda1a8122f1812ec4b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682657846445 2023-04-28 04:57:30,376 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 90cb022f4b644f23a6e36c76ac5baf15, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682657847597 2023-04-28 04:57:30,376 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5e715c5ff5d24beca6a7df4309cd434b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682657847597 2023-04-28 04:57:30,376 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d230ff5bbc5a47ab8b78812d9e89665f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682657849765 2023-04-28 04:57:30,376 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 923de6031e6c4b94afba1f8fea8b5c3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682657849765 2023-04-28 04:57:30,391 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#B#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:30,400 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#A#compaction#321 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:30,417 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:30,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:30,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:30,419 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:30,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:30,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:30,419 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:30,420 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,420 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:30,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:30,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:30,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:30,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:30,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:30,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:30,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:30,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:30,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:30,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:30,433 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. started... 2023-04-28 04:57:30,434 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:30,434 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:30,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:30,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. as already flushing 2023-04-28 04:57:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657910544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657910545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657910555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657910554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55846 deadline: 1682657910631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,636 DEBUG [Thread-2004] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:57:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657910656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657910656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657910662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657910663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,808 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/8263d88c6d0843508faeaba558c1be75 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8263d88c6d0843508faeaba558c1be75 2023-04-28 04:57:30,816 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/B of 44c8ceb2d8e851788f33fa323eec3cf0 into 8263d88c6d0843508faeaba558c1be75(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:30,816 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:30,816 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/B, priority=13, startTime=1682657850374; duration=0sec 2023-04-28 04:57:30,816 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:30,816 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:30,817 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:30,817 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/C is initiating minor compaction (all files) 2023-04-28 04:57:30,817 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/C in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:30,817 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/b609607cbecb4fe190839064cf518923, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/c54d4a759e70469e9ff1efa04980bede, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/3fffbc83303641f58eef2e7bad5ae955] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=36.1 K 2023-04-28 04:57:30,818 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b609607cbecb4fe190839064cf518923, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682657846445 2023-04-28 04:57:30,818 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c54d4a759e70469e9ff1efa04980bede, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682657847597 2023-04-28 04:57:30,818 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 3fffbc83303641f58eef2e7bad5ae955, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682657849765 2023-04-28 04:57:30,827 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#C#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:30,849 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/89074c80f6ef412f8378c2d9e8fa10f5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/89074c80f6ef412f8378c2d9e8fa10f5 2023-04-28 04:57:30,853 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/C of 44c8ceb2d8e851788f33fa323eec3cf0 into 89074c80f6ef412f8378c2d9e8fa10f5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:30,853 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:30,853 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/C, priority=13, startTime=1682657850375; duration=0sec 2023-04-28 04:57:30,853 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:30,861 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a24a79ecdc8d4698bfd25cc330af4c80 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a24a79ecdc8d4698bfd25cc330af4c80 2023-04-28 04:57:30,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657910860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657910860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657910867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:30,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:30,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657910868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,072 DEBUG [Thread-2013] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69fd9cc7 to 127.0.0.1:52932 2023-04-28 04:57:31,072 DEBUG [Thread-2021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4965c362 to 127.0.0.1:52932 2023-04-28 04:57:31,072 DEBUG [Thread-2021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:31,072 DEBUG [Thread-2013] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:31,074 DEBUG [Thread-2015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54da4907 to 127.0.0.1:52932 2023-04-28 04:57:31,074 DEBUG [Thread-2015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:31,078 DEBUG [Thread-2017] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52cea5eb to 127.0.0.1:52932 2023-04-28 04:57:31,078 DEBUG [Thread-2017] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:31,079 DEBUG [Thread-2019] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3546dad6 to 127.0.0.1:52932 2023-04-28 04:57:31,079 DEBUG [Thread-2019] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:31,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657911176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657911176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657911177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657911178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,284 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/91093c044dd94624ab0b3a87dd514f9b 2023-04-28 04:57:31,636 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/3834c7d5b8ae469b9b90b98c571d5ce7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3834c7d5b8ae469b9b90b98c571d5ce7 2023-04-28 04:57:31,643 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/A of 44c8ceb2d8e851788f33fa323eec3cf0 into 3834c7d5b8ae469b9b90b98c571d5ce7(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:57:31,644 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:31,644 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/A, priority=13, startTime=1682657850374; duration=1sec 2023-04-28 04:57:31,644 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:31,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55850 deadline: 1682657911682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55814 deadline: 1682657911682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55810 deadline: 1682657911682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:31,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:55784 deadline: 1682657911683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:31,694 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/c3f6bd18e67a432db044b4fbd16ade36 2023-04-28 04:57:32,111 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/bef6fbb6baf947c6b8012f9271f16157 2023-04-28 04:57:32,114 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/91093c044dd94624ab0b3a87dd514f9b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/91093c044dd94624ab0b3a87dd514f9b 2023-04-28 04:57:32,118 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/91093c044dd94624ab0b3a87dd514f9b, entries=150, sequenceid=303, filesize=25.4 K 2023-04-28 04:57:32,119 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/c3f6bd18e67a432db044b4fbd16ade36 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/c3f6bd18e67a432db044b4fbd16ade36 2023-04-28 04:57:32,123 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/c3f6bd18e67a432db044b4fbd16ade36, entries=150, sequenceid=303, filesize=11.9 K 2023-04-28 04:57:32,123 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/bef6fbb6baf947c6b8012f9271f16157 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/bef6fbb6baf947c6b8012f9271f16157 2023-04-28 04:57:32,127 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/bef6fbb6baf947c6b8012f9271f16157, entries=150, sequenceid=303, filesize=11.9 K 2023-04-28 04:57:32,128 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1694ms, sequenceid=303, compaction requested=false 2023-04-28 04:57:32,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:32,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:32,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:32,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:32,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:32,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:32,194 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,194 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:32,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,195 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f721ff2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:32,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:32,198 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,246 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,247 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:32,247 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:32,247 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:32,247 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,247 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:32,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:32,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:32,341 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:32,341 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:32,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:32,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:32,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:32,349 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:32,349 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:32,350 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:32,350 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:32,350 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:32,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:32,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:32,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:32,555 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,558 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:32,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:57:32,690 DEBUG [Thread-2008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x019e7fff to 127.0.0.1:52932 2023-04-28 04:57:32,690 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:32,691 DEBUG [Thread-2008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:32,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:32,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:32,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:32,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:32,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:32,694 DEBUG [Thread-2006] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39e3a826 to 127.0.0.1:52932 2023-04-28 04:57:32,694 DEBUG [Thread-2002] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28a98302 to 127.0.0.1:52932 2023-04-28 04:57:32,694 DEBUG [Thread-2002] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:32,694 DEBUG [Thread-2006] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:32,694 DEBUG [Thread-2010] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47e30d1c to 127.0.0.1:52932 2023-04-28 04:57:32,694 DEBUG [Thread-2010] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:32,755 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:32,755 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:32,755 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:32,755 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:32,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:32,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:32,755 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:32,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:32,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:32,756 DEBUG [Thread-2012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:32,756 DEBUG [Thread-2012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:32,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:32,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:32,856 DEBUG [Thread-2012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:32,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:32,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:33,113 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b450b84043cc45a19663bd8da7362709 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b450b84043cc45a19663bd8da7362709 2023-04-28 04:57:33,522 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/d1058b3a3f8e40c7869f35c66efdbca1 2023-04-28 04:57:33,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/38723a34a70b45229705bfd06e4b4882 2023-04-28 04:57:34,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/e325e4b3af7c442a88d45765a356ff45 2023-04-28 04:57:34,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/d1058b3a3f8e40c7869f35c66efdbca1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d1058b3a3f8e40c7869f35c66efdbca1 2023-04-28 04:57:34,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d1058b3a3f8e40c7869f35c66efdbca1, entries=150, sequenceid=326, filesize=25.4 K 2023-04-28 04:57:34,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/38723a34a70b45229705bfd06e4b4882 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/38723a34a70b45229705bfd06e4b4882 2023-04-28 04:57:34,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/38723a34a70b45229705bfd06e4b4882, entries=150, sequenceid=326, filesize=11.9 K 2023-04-28 04:57:34,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/e325e4b3af7c442a88d45765a356ff45 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e325e4b3af7c442a88d45765a356ff45 2023-04-28 04:57:34,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e325e4b3af7c442a88d45765a356ff45, entries=150, sequenceid=326, filesize=11.9 K 2023-04-28 04:57:34,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1691ms, sequenceid=326, compaction requested=true 2023-04-28 04:57:34,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:34,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:34,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:34,381 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:34,381 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:34,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:34,382 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78820 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:34,382 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:34,382 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/A is initiating minor compaction (all files) 2023-04-28 04:57:34,382 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/B is initiating minor compaction (all files) 2023-04-28 04:57:34,382 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/A in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:34,382 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/B in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:34,382 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3834c7d5b8ae469b9b90b98c571d5ce7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/91093c044dd94624ab0b3a87dd514f9b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d1058b3a3f8e40c7869f35c66efdbca1] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=77.0 K 2023-04-28 04:57:34,382 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8263d88c6d0843508faeaba558c1be75, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/c3f6bd18e67a432db044b4fbd16ade36, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/38723a34a70b45229705bfd06e4b4882] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=36.4 K 2023-04-28 04:57:34,383 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 3834c7d5b8ae469b9b90b98c571d5ce7, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682657849765 2023-04-28 04:57:34,383 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 8263d88c6d0843508faeaba558c1be75, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682657849765 2023-04-28 04:57:34,383 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 91093c044dd94624ab0b3a87dd514f9b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682657849896 2023-04-28 04:57:34,383 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c3f6bd18e67a432db044b4fbd16ade36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682657849896 2023-04-28 04:57:34,383 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d1058b3a3f8e40c7869f35c66efdbca1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682657850542 2023-04-28 04:57:34,383 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 38723a34a70b45229705bfd06e4b4882, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682657850542 2023-04-28 04:57:34,398 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#B#compaction#329 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:34,407 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#A#compaction#330 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:34,661 DEBUG [Thread-2004] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39d5c003 to 127.0.0.1:52932 2023-04-28 04:57:34,661 DEBUG [Thread-2004] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:34,661 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 04:57:34,661 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1936 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5808 rows 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1941 2023-04-28 04:57:34,662 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5818 rows 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1928 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5784 rows 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1942 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5824 rows 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1936 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5806 rows 2023-04-28 04:57:34,663 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:57:34,663 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29738198 to 127.0.0.1:52932 2023-04-28 04:57:34,663 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:57:34,665 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 04:57:34,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.HMaster$11(2418): Client=jenkins//136.243.104.168 disable TestAcidGuarantees 2023-04-28 04:57:34,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:34,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:34,670 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657854670"}]},"ts":"1682657854670"} 2023-04-28 04:57:34,671 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 04:57:34,695 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 04:57:34,696 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, UNASSIGN}] 2023-04-28 04:57:34,697 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, UNASSIGN 2023-04-28 04:57:34,698 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:34,698 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657854698"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657854698"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657854698"}]},"ts":"1682657854698"} 2023-04-28 04:57:34,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:57:34,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:34,811 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/1bc6fb6ac39047cc9e0305fa62cd9e61 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1bc6fb6ac39047cc9e0305fa62cd9e61 2023-04-28 04:57:34,815 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/B of 44c8ceb2d8e851788f33fa323eec3cf0 into 1bc6fb6ac39047cc9e0305fa62cd9e61(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:34,815 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:34,815 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/B, priority=13, startTime=1682657854381; duration=0sec 2023-04-28 04:57:34,815 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:34,815 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:34,816 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:34,816 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 44c8ceb2d8e851788f33fa323eec3cf0/C is initiating minor compaction (all files) 2023-04-28 04:57:34,816 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44c8ceb2d8e851788f33fa323eec3cf0/C in TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:34,816 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/89074c80f6ef412f8378c2d9e8fa10f5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/bef6fbb6baf947c6b8012f9271f16157, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e325e4b3af7c442a88d45765a356ff45] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp, totalSize=36.4 K 2023-04-28 04:57:34,816 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 89074c80f6ef412f8378c2d9e8fa10f5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682657849765 2023-04-28 04:57:34,817 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting bef6fbb6baf947c6b8012f9271f16157, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682657849896 2023-04-28 04:57:34,817 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting e325e4b3af7c442a88d45765a356ff45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682657850542 2023-04-28 04:57:34,833 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 44c8ceb2d8e851788f33fa323eec3cf0#C#compaction#331 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:34,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:34,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 44c8ceb2d8e851788f33fa323eec3cf0, disabling compactions & flushes 2023-04-28 04:57:34,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:34,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/7061627572c245dbadd2a592eba2284a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/18b9717069354628a224ade76cc6bf88, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/5b7b369125784987967236800399a9eb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/745b7dd795154960a8f1fa4788c4d384, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/abccef5be6b241e8b4a06d962089753b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/893d5fb0ccb849318de98e794153a050, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3a15ccdaa33949379ab89f61ead38c66, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/2fa3c2988cfe4f868c5a8475d874dd50, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/8ad84c4701d045d8ae5129054d61c593, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/67df4a7891e84382bb58cdf4cb5ca637, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/9f3e06a0c0104a429405366812d040fa, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/4d17c4f8ccc94287b0bd817397ee6a63, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/a826b14c21c24271806093657210aa9b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6dbd522618f04c5986baab8d3233f040, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3c7334659dc24c93b9cca2763455b1ec, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/0c5570e0e4de437e9335a75faa4be5ff, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6867156731ac4d499ee109b3bb1c0200, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/90cb022f4b644f23a6e36c76ac5baf15, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d230ff5bbc5a47ab8b78812d9e89665f] to archive 2023-04-28 04:57:34,954 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:34,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/7061627572c245dbadd2a592eba2284a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/7061627572c245dbadd2a592eba2284a 2023-04-28 04:57:34,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/18b9717069354628a224ade76cc6bf88 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/18b9717069354628a224ade76cc6bf88 2023-04-28 04:57:34,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/5b7b369125784987967236800399a9eb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/5b7b369125784987967236800399a9eb 2023-04-28 04:57:34,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ba1038f8ce8d4fb4ac6e55257a7ad2ea 2023-04-28 04:57:34,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/745b7dd795154960a8f1fa4788c4d384 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/745b7dd795154960a8f1fa4788c4d384 2023-04-28 04:57:34,961 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/abccef5be6b241e8b4a06d962089753b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/abccef5be6b241e8b4a06d962089753b 2023-04-28 04:57:34,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/893d5fb0ccb849318de98e794153a050 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/893d5fb0ccb849318de98e794153a050 2023-04-28 04:57:34,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3a15ccdaa33949379ab89f61ead38c66 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3a15ccdaa33949379ab89f61ead38c66 2023-04-28 04:57:34,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/2fa3c2988cfe4f868c5a8475d874dd50 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/2fa3c2988cfe4f868c5a8475d874dd50 2023-04-28 04:57:34,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/8ad84c4701d045d8ae5129054d61c593 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/8ad84c4701d045d8ae5129054d61c593 2023-04-28 04:57:34,965 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/67df4a7891e84382bb58cdf4cb5ca637 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/67df4a7891e84382bb58cdf4cb5ca637 2023-04-28 04:57:34,966 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/9f3e06a0c0104a429405366812d040fa to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/9f3e06a0c0104a429405366812d040fa 2023-04-28 04:57:34,966 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/4d17c4f8ccc94287b0bd817397ee6a63 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/4d17c4f8ccc94287b0bd817397ee6a63 2023-04-28 04:57:34,967 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/a826b14c21c24271806093657210aa9b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/a826b14c21c24271806093657210aa9b 2023-04-28 04:57:34,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6dbd522618f04c5986baab8d3233f040 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6dbd522618f04c5986baab8d3233f040 2023-04-28 04:57:34,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3c7334659dc24c93b9cca2763455b1ec to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3c7334659dc24c93b9cca2763455b1ec 2023-04-28 04:57:34,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/0c5570e0e4de437e9335a75faa4be5ff to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/0c5570e0e4de437e9335a75faa4be5ff 2023-04-28 04:57:34,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6867156731ac4d499ee109b3bb1c0200 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/6867156731ac4d499ee109b3bb1c0200 2023-04-28 04:57:34,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/90cb022f4b644f23a6e36c76ac5baf15 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/90cb022f4b644f23a6e36c76ac5baf15 2023-04-28 04:57:34,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d230ff5bbc5a47ab8b78812d9e89665f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d230ff5bbc5a47ab8b78812d9e89665f 2023-04-28 04:57:34,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:34,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1d9f646f6318442dbdba108b894579d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b8a95ae03d524cfd8c446261238550af, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8140378584464cd09bc8a81b12193308, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/9fd4272d44d54e55842d026161c5748e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b7dab2fd6eaa45c2a9dd26ccdaff18be, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5012a39202804457928a9473a44e8aca, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5bc962ca118b419c86ce1742ec991e71, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/83f2baba1add449abe2f1b709c25688f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d160edafda1f43debaf77629a0e65e05, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8a889d0ea1804410a47e9e49a203ca6b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/08468d72aa1c4cb2b7ce342571a16585, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/a5c6da46257f4308b4f95be138194239, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/16645b748aa74471a653163eef994f09, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/28ccade5d4fe4d93847a67fa56099879, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47d37e77808445679f536f37264b98dc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d12d348cd79f436db5a6b6edc8476250, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/cb8b8cfa4a9640cda1a8122f1812ec4b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/3c1cd980fb5942e8aa9c64968ee09ad7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5e715c5ff5d24beca6a7df4309cd434b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8263d88c6d0843508faeaba558c1be75, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/923de6031e6c4b94afba1f8fea8b5c3f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/c3f6bd18e67a432db044b4fbd16ade36, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/38723a34a70b45229705bfd06e4b4882] to archive 2023-04-28 04:57:34,976 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:34,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1d9f646f6318442dbdba108b894579d3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1d9f646f6318442dbdba108b894579d3 2023-04-28 04:57:34,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b8a95ae03d524cfd8c446261238550af to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b8a95ae03d524cfd8c446261238550af 2023-04-28 04:57:34,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8140378584464cd09bc8a81b12193308 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8140378584464cd09bc8a81b12193308 2023-04-28 04:57:34,983 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/9fd4272d44d54e55842d026161c5748e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/9fd4272d44d54e55842d026161c5748e 2023-04-28 04:57:34,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b7dab2fd6eaa45c2a9dd26ccdaff18be to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/b7dab2fd6eaa45c2a9dd26ccdaff18be 2023-04-28 04:57:34,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5012a39202804457928a9473a44e8aca to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5012a39202804457928a9473a44e8aca 2023-04-28 04:57:34,988 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5bc962ca118b419c86ce1742ec991e71 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5bc962ca118b419c86ce1742ec991e71 2023-04-28 04:57:34,990 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/83f2baba1add449abe2f1b709c25688f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/83f2baba1add449abe2f1b709c25688f 2023-04-28 04:57:34,991 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d160edafda1f43debaf77629a0e65e05 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d160edafda1f43debaf77629a0e65e05 2023-04-28 04:57:34,992 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8a889d0ea1804410a47e9e49a203ca6b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8a889d0ea1804410a47e9e49a203ca6b 2023-04-28 04:57:34,992 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/08468d72aa1c4cb2b7ce342571a16585 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/08468d72aa1c4cb2b7ce342571a16585 2023-04-28 04:57:34,993 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/a5c6da46257f4308b4f95be138194239 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/a5c6da46257f4308b4f95be138194239 2023-04-28 04:57:34,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/16645b748aa74471a653163eef994f09 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/16645b748aa74471a653163eef994f09 2023-04-28 04:57:34,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/28ccade5d4fe4d93847a67fa56099879 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/28ccade5d4fe4d93847a67fa56099879 2023-04-28 04:57:34,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47d37e77808445679f536f37264b98dc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47d37e77808445679f536f37264b98dc 2023-04-28 04:57:34,996 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d12d348cd79f436db5a6b6edc8476250 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/d12d348cd79f436db5a6b6edc8476250 2023-04-28 04:57:34,997 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/cb8b8cfa4a9640cda1a8122f1812ec4b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/cb8b8cfa4a9640cda1a8122f1812ec4b 2023-04-28 04:57:34,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/3c1cd980fb5942e8aa9c64968ee09ad7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/3c1cd980fb5942e8aa9c64968ee09ad7 2023-04-28 04:57:34,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5e715c5ff5d24beca6a7df4309cd434b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/5e715c5ff5d24beca6a7df4309cd434b 2023-04-28 04:57:34,999 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8263d88c6d0843508faeaba558c1be75 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/8263d88c6d0843508faeaba558c1be75 2023-04-28 04:57:35,000 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/923de6031e6c4b94afba1f8fea8b5c3f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/923de6031e6c4b94afba1f8fea8b5c3f 2023-04-28 04:57:35,001 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/c3f6bd18e67a432db044b4fbd16ade36 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/c3f6bd18e67a432db044b4fbd16ade36 2023-04-28 04:57:35,002 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/38723a34a70b45229705bfd06e4b4882 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/38723a34a70b45229705bfd06e4b4882 2023-04-28 04:57:35,007 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4dc6872c3fc04b42a1657a590c5bdcea, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4671e81055ef4e13ad5cac6cd75f6004, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/663617b983e5435fa6c66824a8c791fe, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/f444002247dc45bf8ef5d4468eba3ebb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/9f47bbb57d374a8e8756fe6b06a7a8d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/762abe80cc5b4dc18f2b431f2c3b8ab6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e562379d3936459398a755f997057291, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/2da2734b1bc846a09bd2f963341d9754, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/1162e94e6bc14c04977f13b8678c4fd3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0f197484eb544909b16c73cc052b52b8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/a20eec15d1c04b179b65cb7c89444da9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7ffa602b63414ae784df96d6a97d30c4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/328b0e2eb1ef43a08f14ae40151b6fde, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7095953fd8154630b45449496f14f2c8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9bc4c36ced47fc8eb040b3e2f4340b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0c69fc0ec3984c3cbe46ae1593219aef, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/b609607cbecb4fe190839064cf518923, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/87c074fdabf44e55884bef2a4701066f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/c54d4a759e70469e9ff1efa04980bede, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/3fffbc83303641f58eef2e7bad5ae955] to archive 2023-04-28 04:57:35,008 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:35,009 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4dc6872c3fc04b42a1657a590c5bdcea to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4dc6872c3fc04b42a1657a590c5bdcea 2023-04-28 04:57:35,010 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4671e81055ef4e13ad5cac6cd75f6004 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/4671e81055ef4e13ad5cac6cd75f6004 2023-04-28 04:57:35,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/663617b983e5435fa6c66824a8c791fe to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/663617b983e5435fa6c66824a8c791fe 2023-04-28 04:57:35,012 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/f444002247dc45bf8ef5d4468eba3ebb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/f444002247dc45bf8ef5d4468eba3ebb 2023-04-28 04:57:35,013 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/9f47bbb57d374a8e8756fe6b06a7a8d3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/9f47bbb57d374a8e8756fe6b06a7a8d3 2023-04-28 04:57:35,014 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/762abe80cc5b4dc18f2b431f2c3b8ab6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/762abe80cc5b4dc18f2b431f2c3b8ab6 2023-04-28 04:57:35,015 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e562379d3936459398a755f997057291 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e562379d3936459398a755f997057291 2023-04-28 04:57:35,016 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/2da2734b1bc846a09bd2f963341d9754 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/2da2734b1bc846a09bd2f963341d9754 2023-04-28 04:57:35,018 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/1162e94e6bc14c04977f13b8678c4fd3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/1162e94e6bc14c04977f13b8678c4fd3 2023-04-28 04:57:35,019 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0f197484eb544909b16c73cc052b52b8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0f197484eb544909b16c73cc052b52b8 2023-04-28 04:57:35,019 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/a20eec15d1c04b179b65cb7c89444da9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/a20eec15d1c04b179b65cb7c89444da9 2023-04-28 04:57:35,021 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7ffa602b63414ae784df96d6a97d30c4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7ffa602b63414ae784df96d6a97d30c4 2023-04-28 04:57:35,021 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/328b0e2eb1ef43a08f14ae40151b6fde to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/328b0e2eb1ef43a08f14ae40151b6fde 2023-04-28 04:57:35,022 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7095953fd8154630b45449496f14f2c8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/7095953fd8154630b45449496f14f2c8 2023-04-28 04:57:35,023 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9bc4c36ced47fc8eb040b3e2f4340b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9bc4c36ced47fc8eb040b3e2f4340b 2023-04-28 04:57:35,024 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0c69fc0ec3984c3cbe46ae1593219aef to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/0c69fc0ec3984c3cbe46ae1593219aef 2023-04-28 04:57:35,025 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/b609607cbecb4fe190839064cf518923 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/b609607cbecb4fe190839064cf518923 2023-04-28 04:57:35,026 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/87c074fdabf44e55884bef2a4701066f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/87c074fdabf44e55884bef2a4701066f 2023-04-28 04:57:35,027 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/c54d4a759e70469e9ff1efa04980bede to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/c54d4a759e70469e9ff1efa04980bede 2023-04-28 04:57:35,027 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase12:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/3fffbc83303641f58eef2e7bad5ae955 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/3fffbc83303641f58eef2e7bad5ae955 2023-04-28 04:57:35,239 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/ebd4910b16464faab88c782d6c2412c1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ebd4910b16464faab88c782d6c2412c1 2023-04-28 04:57:35,244 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/A of 44c8ceb2d8e851788f33fa323eec3cf0 into ebd4910b16464faab88c782d6c2412c1(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:35,244 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:35,244 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/A, priority=13, startTime=1682657854381; duration=0sec 2023-04-28 04:57:35,244 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:35,246 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/5f47abb87285441d8d3612511e03189f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/5f47abb87285441d8d3612511e03189f 2023-04-28 04:57:35,250 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44c8ceb2d8e851788f33fa323eec3cf0/C of 44c8ceb2d8e851788f33fa323eec3cf0 into 5f47abb87285441d8d3612511e03189f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:35,250 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:35,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. after waiting 0 ms 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:35,250 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0., storeName=44c8ceb2d8e851788f33fa323eec3cf0/C, priority=13, startTime=1682657854381; duration=0sec 2023-04-28 04:57:35,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing 44c8ceb2d8e851788f33fa323eec3cf0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 04:57:35,250 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=A 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=B 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44c8ceb2d8e851788f33fa323eec3cf0, store=C 2023-04-28 04:57:35,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:35,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283bf2dc452dca4f299a8ac79abf4877e0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283bf2dc452dca4f299a8ac79abf4877e0 2023-04-28 04:57:35,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:36,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/44a09f5f40b04809a4f0066ec5c26405 2023-04-28 04:57:36,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/47a2dc6282114f73b2b4d91c918149e0 2023-04-28 04:57:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:36,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/da9463001fd54b64a7a30e2a07e2852f 2023-04-28 04:57:36,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/A/44a09f5f40b04809a4f0066ec5c26405 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/44a09f5f40b04809a4f0066ec5c26405 2023-04-28 04:57:36,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/44a09f5f40b04809a4f0066ec5c26405, entries=100, sequenceid=336, filesize=18.6 K 2023-04-28 04:57:36,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/B/47a2dc6282114f73b2b4d91c918149e0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47a2dc6282114f73b2b4d91c918149e0 2023-04-28 04:57:36,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47a2dc6282114f73b2b4d91c918149e0, entries=100, sequenceid=336, filesize=9.5 K 2023-04-28 04:57:36,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/.tmp/C/da9463001fd54b64a7a30e2a07e2852f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9463001fd54b64a7a30e2a07e2852f 2023-04-28 04:57:36,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9463001fd54b64a7a30e2a07e2852f, entries=100, sequenceid=336, filesize=9.5 K 2023-04-28 04:57:36,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 44c8ceb2d8e851788f33fa323eec3cf0 in 1679ms, sequenceid=336, compaction requested=false 2023-04-28 04:57:36,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3834c7d5b8ae469b9b90b98c571d5ce7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/91093c044dd94624ab0b3a87dd514f9b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d1058b3a3f8e40c7869f35c66efdbca1] to archive 2023-04-28 04:57:36,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:36,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3834c7d5b8ae469b9b90b98c571d5ce7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/3834c7d5b8ae469b9b90b98c571d5ce7 2023-04-28 04:57:36,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/91093c044dd94624ab0b3a87dd514f9b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/91093c044dd94624ab0b3a87dd514f9b 2023-04-28 04:57:36,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d1058b3a3f8e40c7869f35c66efdbca1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/d1058b3a3f8e40c7869f35c66efdbca1 2023-04-28 04:57:36,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/89074c80f6ef412f8378c2d9e8fa10f5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/bef6fbb6baf947c6b8012f9271f16157, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e325e4b3af7c442a88d45765a356ff45] to archive 2023-04-28 04:57:36,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:57:36,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/89074c80f6ef412f8378c2d9e8fa10f5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/89074c80f6ef412f8378c2d9e8fa10f5 2023-04-28 04:57:36,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/bef6fbb6baf947c6b8012f9271f16157 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/bef6fbb6baf947c6b8012f9271f16157 2023-04-28 04:57:36,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e325e4b3af7c442a88d45765a356ff45 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/e325e4b3af7c442a88d45765a356ff45 2023-04-28 04:57:36,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/recovered.edits/339.seqid, newMaxSeqId=339, maxSeqId=4 2023-04-28 04:57:36,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0. 2023-04-28 04:57:36,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 44c8ceb2d8e851788f33fa323eec3cf0: 2023-04-28 04:57:36,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed 44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:36,957 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=44c8ceb2d8e851788f33fa323eec3cf0, regionState=CLOSED 2023-04-28 04:57:36,957 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657856957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657856957"}]},"ts":"1682657856957"} 2023-04-28 04:57:36,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-28 04:57:36,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 44c8ceb2d8e851788f33fa323eec3cf0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 2.2600 sec 2023-04-28 04:57:36,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-28 04:57:36,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=44c8ceb2d8e851788f33fa323eec3cf0, UNASSIGN in 2.2640 sec 2023-04-28 04:57:36,963 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657856963"}]},"ts":"1682657856963"} 2023-04-28 04:57:36,964 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 04:57:37,011 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 04:57:37,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.3460 sec 2023-04-28 04:57:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 04:57:38,775 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-28 04:57:38,775 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$5(2228): Client=jenkins//136.243.104.168 delete TestAcidGuarantees 2023-04-28 04:57:38,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:38,777 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:38,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 04:57:38,778 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:38,783 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:38,786 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/recovered.edits] 2023-04-28 04:57:38,791 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/44a09f5f40b04809a4f0066ec5c26405 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/44a09f5f40b04809a4f0066ec5c26405 2023-04-28 04:57:38,792 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ebd4910b16464faab88c782d6c2412c1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/A/ebd4910b16464faab88c782d6c2412c1 2023-04-28 04:57:38,795 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1bc6fb6ac39047cc9e0305fa62cd9e61 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/1bc6fb6ac39047cc9e0305fa62cd9e61 2023-04-28 04:57:38,796 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47a2dc6282114f73b2b4d91c918149e0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/B/47a2dc6282114f73b2b4d91c918149e0 2023-04-28 04:57:38,799 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/5f47abb87285441d8d3612511e03189f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/5f47abb87285441d8d3612511e03189f 2023-04-28 04:57:38,801 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9463001fd54b64a7a30e2a07e2852f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/C/da9463001fd54b64a7a30e2a07e2852f 2023-04-28 04:57:38,805 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/recovered.edits/339.seqid to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0/recovered.edits/339.seqid 2023-04-28 04:57:38,806 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/44c8ceb2d8e851788f33fa323eec3cf0 2023-04-28 04:57:38,806 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:57:38,806 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 04:57:38,808 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 04:57:38,812 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281a22dcdfdcd14a73bf23e3d38bfe3e5b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281a22dcdfdcd14a73bf23e3d38bfe3e5b 2023-04-28 04:57:38,814 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042825568dc837dc426a8ed3bf684946ce6e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042825568dc837dc426a8ed3bf684946ce6e 2023-04-28 04:57:38,817 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042833c4f3ff349c43f7a557ba3b631b807f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042833c4f3ff349c43f7a557ba3b631b807f 2023-04-28 04:57:38,819 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283bf2dc452dca4f299a8ac79abf4877e0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283bf2dc452dca4f299a8ac79abf4877e0 2023-04-28 04:57:38,822 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042846a2c5845e5c460aafd3cc5e5a7fef3f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042846a2c5845e5c460aafd3cc5e5a7fef3f 2023-04-28 04:57:38,824 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284bc13fb7280340a5bfe1bed6b6355b25 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304284bc13fb7280340a5bfe1bed6b6355b25 2023-04-28 04:57:38,826 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428783d83ec39794815b208c76bb15ddc0f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428783d83ec39794815b208c76bb15ddc0f 2023-04-28 04:57:38,828 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289f793e98e0894f6daea2fafd3dea8207 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289f793e98e0894f6daea2fafd3dea8207 2023-04-28 04:57:38,829 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a24a79ecdc8d4698bfd25cc330af4c80 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a24a79ecdc8d4698bfd25cc330af4c80 2023-04-28 04:57:38,831 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a49dd711b6ff496e9b9f0a24a82c53ad to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a49dd711b6ff496e9b9f0a24a82c53ad 2023-04-28 04:57:38,833 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a6190319f9e0410e80ee10f7fd1b3bf2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a6190319f9e0410e80ee10f7fd1b3bf2 2023-04-28 04:57:38,835 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ad63888fb9cc4104b21a3da0ecb6bfa8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ad63888fb9cc4104b21a3da0ecb6bfa8 2023-04-28 04:57:38,837 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b450b84043cc45a19663bd8da7362709 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b450b84043cc45a19663bd8da7362709 2023-04-28 04:57:38,839 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428be4314cd85d9483b8d9ff471c21958f1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428be4314cd85d9483b8d9ff471c21958f1 2023-04-28 04:57:38,840 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c6e524a40fd34098841f02bd3d587297 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c6e524a40fd34098841f02bd3d587297 2023-04-28 04:57:38,842 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cad2458334b34f8783fbe00c0d5db947 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cad2458334b34f8783fbe00c0d5db947 2023-04-28 04:57:38,844 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cce48a8b96524361aa29c74bfcc2aee0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cce48a8b96524361aa29c74bfcc2aee0 2023-04-28 04:57:38,846 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ddca5a8de09d4704b4a89aba08619e5a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ddca5a8de09d4704b4a89aba08619e5a 2023-04-28 04:57:38,848 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ea26e872de1a4db7936b83eaa9cb86ab to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ea26e872de1a4db7936b83eaa9cb86ab 2023-04-28 04:57:38,848 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 04:57:38,850 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:38,852 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 04:57:38,854 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 04:57:38,855 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:38,855 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 04:57:38,855 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682657858855"}]},"ts":"9223372036854775807"} 2023-04-28 04:57:38,857 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 04:57:38,857 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 44c8ceb2d8e851788f33fa323eec3cf0, NAME => 'TestAcidGuarantees,,1682657828854.44c8ceb2d8e851788f33fa323eec3cf0.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 04:57:38,857 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 04:57:38,857 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682657858857"}]},"ts":"9223372036854775807"} 2023-04-28 04:57:38,859 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 04:57:38,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 04:57:38,906 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:38,908 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 131 msec 2023-04-28 04:57:39,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 04:57:39,079 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-28 04:57:39,088 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=527 (was 540), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=497 (was 476) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=9965 (was 10264) 2023-04-28 04:57:39,098 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=527, MaxFileDescriptor=60000, SystemLoadAverage=497, ProcessCount=171, AvailableMemoryMB=9965 2023-04-28 04:57:39,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:57:39,100 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$4(2112): Client=jenkins//136.243.104.168 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:57:39,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 04:57:39,102 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:57:39,103 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(697): Client=jenkins//136.243.104.168 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-28 04:57:39,103 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:57:39,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 04:57:39,104 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,105 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 empty. 2023-04-28 04:57:39,105 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,105 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:57:39,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 04:57:39,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 04:57:39,512 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:57:39,517 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 51e65c0b85e192dace926daeb81e88b4, NAME => 'TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:57:39,531 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:57:39,531 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 51e65c0b85e192dace926daeb81e88b4, disabling compactions & flushes 2023-04-28 04:57:39,531 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,531 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,531 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. after waiting 0 ms 2023-04-28 04:57:39,531 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,531 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,531 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:39,533 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:57:39,534 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657859534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657859534"}]},"ts":"1682657859534"} 2023-04-28 04:57:39,535 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:57:39,535 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:57:39,535 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657859535"}]},"ts":"1682657859535"} 2023-04-28 04:57:39,536 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 04:57:39,600 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, ASSIGN}] 2023-04-28 04:57:39,603 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, ASSIGN 2023-04-28 04:57:39,604 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:57:39,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 04:57:39,755 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=51e65c0b85e192dace926daeb81e88b4, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:39,755 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657859755"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657859755"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657859755"}]},"ts":"1682657859755"} 2023-04-28 04:57:39,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:57:39,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 51e65c0b85e192dace926daeb81e88b4, NAME => 'TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:57:39,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:57:39,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,911 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,912 DEBUG [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A 2023-04-28 04:57:39,912 DEBUG [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A 2023-04-28 04:57:39,912 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:39,912 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e65c0b85e192dace926daeb81e88b4 columnFamilyName A 2023-04-28 04:57:39,913 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.HStore(310): Store=51e65c0b85e192dace926daeb81e88b4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:39,913 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,914 DEBUG [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B 2023-04-28 04:57:39,914 DEBUG [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B 2023-04-28 04:57:39,914 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:39,914 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e65c0b85e192dace926daeb81e88b4 columnFamilyName B 2023-04-28 04:57:39,915 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.HStore(310): Store=51e65c0b85e192dace926daeb81e88b4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:39,915 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,916 DEBUG [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C 2023-04-28 04:57:39,916 DEBUG [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C 2023-04-28 04:57:39,916 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:57:39,916 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e65c0b85e192dace926daeb81e88b4 columnFamilyName C 2023-04-28 04:57:39,917 INFO [StoreOpener-51e65c0b85e192dace926daeb81e88b4-1] regionserver.HStore(310): Store=51e65c0b85e192dace926daeb81e88b4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:57:39,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:57:39,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:39,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:57:39,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened 51e65c0b85e192dace926daeb81e88b4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59529541, jitterRate=-0.112940713763237}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:57:39,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:39,926 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., pid=49, masterSystemTime=1682657859907 2023-04-28 04:57:39,927 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:39,928 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=51e65c0b85e192dace926daeb81e88b4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:39,928 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657859928"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657859928"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657859928"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657859928"}]},"ts":"1682657859928"} 2023-04-28 04:57:39,931 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-28 04:57:39,931 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 in 173 msec 2023-04-28 04:57:39,933 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-28 04:57:39,933 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, ASSIGN in 331 msec 2023-04-28 04:57:39,934 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:57:39,934 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657859934"}]},"ts":"1682657859934"} 2023-04-28 04:57:39,935 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 04:57:39,950 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:57:39,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 850 msec 2023-04-28 04:57:40,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 04:57:40,207 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-28 04:57:40,208 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x02e8c6f6 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,232 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a7ac4a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,233 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,236 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,237 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:57:40,240 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:37076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:57:40,241 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x2cb7a788 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,254 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@682c0b9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,255 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x41ccedcb to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,277 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50d6899d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,277 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x40e6f624 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,306 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6662e946, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,307 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5d7fcf to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,337 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d7deb76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,338 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x4ea9d51a to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,348 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f4b7807, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,349 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x31fc0e3f to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,380 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6523d347, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,380 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x79db57d4 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,399 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28072326, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,399 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x0e56ea94 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,411 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d90797d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,412 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x4486b3c7 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,431 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d3df389, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,431 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x149f1de0 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:57:40,443 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65f69c3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:57:40,452 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:40,461 DEBUG [hconnection-0x70074182-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,461 DEBUG [hconnection-0x117641be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,464 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,464 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,472 DEBUG [hconnection-0x3c985a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,472 DEBUG [hconnection-0x72678ea9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,477 DEBUG [hconnection-0x178d2f9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,478 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,480 DEBUG [hconnection-0x7f0d8c29-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,480 DEBUG [hconnection-0x26e28ef3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:40,496 DEBUG [hconnection-0x4150e0b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,496 DEBUG [hconnection-0x34f40ca3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:40,498 DEBUG [hconnection-0x69258fc3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:57:40,498 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:40,498 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,498 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,500 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,500 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,499 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:40,501 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:40,501 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:40,500 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,503 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,503 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,503 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,504 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:57246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:57:40,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:40,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:57:40,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:40,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:40,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:40,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:40,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:40,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:40,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657860537, max timestamp=1682657860543Num uniques 859; 2023-04-28 04:57:40,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:40,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657860537, max timestamp=1682657860543Num uniques 859; 2023-04-28 04:57:40,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:40,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657860537, max timestamp=1682657860543Num uniques 859; 2023-04-28 04:57:40,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:40,547 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:40,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:40,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:40,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:40,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:40,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:40,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:40,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:40,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:40,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:40,568 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:40,570 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:40,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3066902d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:40,570 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657920568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657920572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657920573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657920574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657920575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,581 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:40,581 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:40,581 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:40,581 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,581 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:40,590 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,590 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:40,590 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:40,590 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:40,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:40,595 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:40,595 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:40,595 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:40,595 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:40,595 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:40,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:40,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:40,611 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:40,611 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:40,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:40,611 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:40,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:40,611 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:40,611 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:40,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:40,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657920675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657920675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657920675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657920676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657920676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,711 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:40,726 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:40,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:40,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:40,736 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:40,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:40,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:40,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:40,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,747 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:40,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:40,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:40,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:40,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:40,753 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:40,754 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:40,754 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:40,759 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:40,759 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:40,759 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:40,759 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:40,759 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:40,768 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,770 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:40,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bc0d75a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:40,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:40,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,779 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,779 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:40,779 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:40,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:40,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:40,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:40,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:40,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:40,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:40,791 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:40,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:40,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:40,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:40,791 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:40,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:40,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,802 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:40,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:40,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:40,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:40,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:40,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:40,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:40,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:40,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:40,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:40,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,811 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:40,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,811 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:40,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657920877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657920877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657920877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657920877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:40,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657920878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,911 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:40,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:40,914 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:40,916 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:40,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:40,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:40,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:40,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,917 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:40,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:40,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:40,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:40,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:40,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:40,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:40,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:40,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:40,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:40,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:40,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:40,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:40,928 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:40,928 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:40,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:40,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:40,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:40,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/337c0572728b41c4b961fdec3ecc71f2 2023-04-28 04:57:40,938 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,938 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:40,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c1423e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:40,940 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:40,940 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:40,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:40,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:40,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:40,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:40,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:40,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:40,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:40,959 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:40,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:40,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:40,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:40,960 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:40,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:40,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:40,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,971 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:40,979 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:40,979 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:40,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:40,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:40,979 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:40,979 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:40,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:40,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,079 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:41,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:41,081 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:41,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:41,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:41,082 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:41,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:41,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:41,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:41,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,082 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,095 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:41,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:41,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:41,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:41,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:41,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:41,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:41,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:41,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:41,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:41,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,107 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:41,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:41,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe35e5b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:41,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,109 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:41,149 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,150 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:41,150 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:41,150 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,150 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:41,150 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:41,169 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:41,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:41,170 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:41,172 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:41,172 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:41,172 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:41,172 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:41,172 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:41,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:41,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,181 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657921183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657921183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657921183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657921183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657921185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:41,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,189 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:41,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:41,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:41,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,189 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:41,190 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:41,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:41,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:41,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,190 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:41,190 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:41,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,190 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,290 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:41,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:41,292 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:41,294 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:41,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:41,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:41,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:41,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:41,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:41,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:41,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:41,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:41,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:41,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:41,306 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:41,307 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:41,307 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:41,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:41,317 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:41,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b004a72[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:41,317 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,326 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:41,326 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:41,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:41,326 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:41,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:41,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:41,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:41,339 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:41,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:41,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:41,339 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:41,339 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:41,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,349 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/a4114407c6fb491cab475b9375fbe853 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:41,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:41,358 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:41,358 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,358 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:41,358 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:41,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/89e76aad9f8b4b858aba3595f0e1a5f6 2023-04-28 04:57:41,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/337c0572728b41c4b961fdec3ecc71f2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/337c0572728b41c4b961fdec3ecc71f2 2023-04-28 04:57:41,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/337c0572728b41c4b961fdec3ecc71f2, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 04:57:41,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/a4114407c6fb491cab475b9375fbe853 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a4114407c6fb491cab475b9375fbe853 2023-04-28 04:57:41,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a4114407c6fb491cab475b9375fbe853, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 04:57:41,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/89e76aad9f8b4b858aba3595f0e1a5f6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/89e76aad9f8b4b858aba3595f0e1a5f6 2023-04-28 04:57:41,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/89e76aad9f8b4b858aba3595f0e1a5f6, entries=150, sequenceid=11, filesize=11.6 K 2023-04-28 04:57:41,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51e65c0b85e192dace926daeb81e88b4 in 863ms, sequenceid=11, compaction requested=false 2023-04-28 04:57:41,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:41,458 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:41,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:41,460 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:41,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:41,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:41,461 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:41,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:41,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:41,461 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:41,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:41,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:41,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:41,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:41,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:41,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:41,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:41,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:41,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:41,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:41,475 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:41,475 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:41,475 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:41,475 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 04:57:41,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:41,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:41,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:41,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:41,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:41,476 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:41,695 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:41,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657921696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657921697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657921698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657921701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657921702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657921804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657921810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657921810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657921810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657921810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,881 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/5e260ab6f1f74242b7ccf8df3905f6a8 2023-04-28 04:57:41,889 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/c718258e522840d1bad8ef55a74e9d51 2023-04-28 04:57:41,903 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/f44f3e4073244c31bc6a517504e88d37 2023-04-28 04:57:41,906 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/5e260ab6f1f74242b7ccf8df3905f6a8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5e260ab6f1f74242b7ccf8df3905f6a8 2023-04-28 04:57:41,913 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5e260ab6f1f74242b7ccf8df3905f6a8, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 04:57:41,913 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/c718258e522840d1bad8ef55a74e9d51 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/c718258e522840d1bad8ef55a74e9d51 2023-04-28 04:57:41,917 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/c718258e522840d1bad8ef55a74e9d51, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 04:57:41,917 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/f44f3e4073244c31bc6a517504e88d37 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f44f3e4073244c31bc6a517504e88d37 2023-04-28 04:57:41,922 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f44f3e4073244c31bc6a517504e88d37, entries=150, sequenceid=48, filesize=11.6 K 2023-04-28 04:57:41,923 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 51e65c0b85e192dace926daeb81e88b4 in 448ms, sequenceid=48, compaction requested=false 2023-04-28 04:57:41,923 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:41,923 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:41,923 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:41,923 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:41,923 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,923 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:41,923 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:41,940 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,940 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,940 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,940 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:41,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36173c4f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:41,942 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:41,942 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:41,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:41,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:41,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:41,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:41,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:41,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:41,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:41,960 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:41,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:41,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:41,960 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:41,960 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:41,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:41,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:41,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:41,969 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,971 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:41,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:41,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:41,979 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:41,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:41,979 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:41,980 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:41,980 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:41,979 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:41,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:42,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:57:42,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:42,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:42,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:42,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:42,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:42,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:42,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657862031, max timestamp=1682657862036Num uniques 859; 2023-04-28 04:57:42,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:57:42,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657862031, max timestamp=1682657862036Num uniques 859; 2023-04-28 04:57:42,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:57:42,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657862031, max timestamp=1682657862036Num uniques 859; 2023-04-28 04:57:42,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:57:42,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657922054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657922055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657922056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657922056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657922061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,080 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:42,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:42,082 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:42,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:42,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:42,083 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:42,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:42,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:42,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:42,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:42,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:42,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:42,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:42,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:42,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:42,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:42,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:42,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3d5801[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:42,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,116 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,116 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:42,116 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:42,116 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:42,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:42,152 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,152 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:42,152 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,152 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:42,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:42,155 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:42,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:42,156 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:42,156 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:42,156 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:42,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657922163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657922163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657922163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657922164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657922164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,177 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,177 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,177 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,177 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,178 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,189 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,190 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:42,190 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:42,190 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:42,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,190 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,191 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:42,290 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:42,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:42,292 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:42,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:42,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:42,294 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:42,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:42,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:42,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:42,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:42,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:42,330 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:42,330 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,330 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:42,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:42,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:42,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:42,344 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:42,344 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:42,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,356 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,356 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,357 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:42,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@969d65b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:42,361 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:42,361 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,369 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,369 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,369 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:42,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:42,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:42,369 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:42,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657922370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657922370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:42,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:42,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657922373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657922373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657922373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:42,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:42,382 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:42,382 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:42,383 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:42,383 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:42,383 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:42,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,390 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,392 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:42,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:42,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,401 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,401 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:42,401 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:42,401 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,401 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/d66ee2e1a1b140a39c9dad4643fa6dcf 2023-04-28 04:57:42,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/947adc8f73c94dfabfcf9b3f66e8296b 2023-04-28 04:57:42,501 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:57:42,501 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:42,503 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:42,508 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:42,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:42,508 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:42,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,509 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,516 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,516 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,516 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,516 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:42,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:42,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:42,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:42,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,517 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:42,524 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:42,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:42,525 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,525 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:42,525 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:42,525 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:42,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:42,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:42,527 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:42,537 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:42,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:42,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50634ede[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:42,539 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:42,539 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,547 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,547 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:42,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,547 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:42,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:42,547 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:42,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:42,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:42,562 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:42,562 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:42,562 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:42,562 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:42,562 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:42,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,570 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,579 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,579 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:42,579 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:42,579 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:42,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,579 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:42,579 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:42,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,580 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,679 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:42,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:42,681 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:42,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:42,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:42,683 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:42,683 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,683 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:42,683 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:42,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657922681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,684 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,684 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657922682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657922682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657922683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:42,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657922684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,695 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:42,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:42,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:42,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:42,696 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,696 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:42,696 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:42,696 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:42,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:42,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:42,706 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f19e5d9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:42,706 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:42,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:42,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:42,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:42,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:42,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:42,728 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:42,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:42,730 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:42,730 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:42,730 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:42,730 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:42,730 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:42,737 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,737 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,737 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:42,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,742 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:42,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:42,806 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:42,806 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:42,806 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:42,806 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:42,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:42,807 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:42,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:42,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/38182ce692934f2c8984db3b9cc85a5a 2023-04-28 04:57:42,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/d66ee2e1a1b140a39c9dad4643fa6dcf as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d66ee2e1a1b140a39c9dad4643fa6dcf 2023-04-28 04:57:42,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d66ee2e1a1b140a39c9dad4643fa6dcf, entries=200, sequenceid=58, filesize=13.9 K 2023-04-28 04:57:42,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/947adc8f73c94dfabfcf9b3f66e8296b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/947adc8f73c94dfabfcf9b3f66e8296b 2023-04-28 04:57:42,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/947adc8f73c94dfabfcf9b3f66e8296b, entries=150, sequenceid=58, filesize=11.6 K 2023-04-28 04:57:42,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/38182ce692934f2c8984db3b9cc85a5a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/38182ce692934f2c8984db3b9cc85a5a 2023-04-28 04:57:42,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/38182ce692934f2c8984db3b9cc85a5a, entries=150, sequenceid=58, filesize=11.6 K 2023-04-28 04:57:42,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 51e65c0b85e192dace926daeb81e88b4 in 878ms, sequenceid=58, compaction requested=true 2023-04-28 04:57:42,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:42,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:42,888 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:42,888 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:42,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:42,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:42,889 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:42,889 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:42,889 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:42,889 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:42,889 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,889 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,889 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/337c0572728b41c4b961fdec3ecc71f2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5e260ab6f1f74242b7ccf8df3905f6a8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d66ee2e1a1b140a39c9dad4643fa6dcf] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=37.1 K 2023-04-28 04:57:42,889 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a4114407c6fb491cab475b9375fbe853, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/c718258e522840d1bad8ef55a74e9d51, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/947adc8f73c94dfabfcf9b3f66e8296b] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=34.8 K 2023-04-28 04:57:42,890 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a4114407c6fb491cab475b9375fbe853, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682657860504 2023-04-28 04:57:42,890 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 337c0572728b41c4b961fdec3ecc71f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682657860504 2023-04-28 04:57:42,890 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c718258e522840d1bad8ef55a74e9d51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682657860571 2023-04-28 04:57:42,890 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5e260ab6f1f74242b7ccf8df3905f6a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682657860571 2023-04-28 04:57:42,890 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 947adc8f73c94dfabfcf9b3f66e8296b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682657861694 2023-04-28 04:57:42,890 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d66ee2e1a1b140a39c9dad4643fa6dcf, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682657861694 2023-04-28 04:57:42,895 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#345 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:42,896 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#344 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:42,906 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:42,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:42,908 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:42,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:42,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:42,913 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:42,913 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:42,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:42,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,924 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/20e4a1f56c3c420594c077b1f1d5fab4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/20e4a1f56c3c420594c077b1f1d5fab4 2023-04-28 04:57:42,925 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/0480db5177524d37bea328734f57f8fb as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0480db5177524d37bea328734f57f8fb 2023-04-28 04:57:42,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:42,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:42,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:42,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:42,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:42,929 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into 0480db5177524d37bea328734f57f8fb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:42,929 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:42,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:42,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:42,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:42,929 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=13, startTime=1682657862888; duration=0sec 2023-04-28 04:57:42,929 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:42,930 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:42,930 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:42,930 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:42,930 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:42,930 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:42,930 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,930 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:42,930 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into 20e4a1f56c3c420594c077b1f1d5fab4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:42,930 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 04:57:42,930 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:42,931 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=13, startTime=1682657862888; duration=0sec 2023-04-28 04:57:42,931 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:42,931 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:42,931 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:42,931 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:42,931 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:42,931 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:42,931 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:42,931 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:42,931 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/89e76aad9f8b4b858aba3595f0e1a5f6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f44f3e4073244c31bc6a517504e88d37, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/38182ce692934f2c8984db3b9cc85a5a] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=34.8 K 2023-04-28 04:57:42,931 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:42,931 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:42,931 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 89e76aad9f8b4b858aba3595f0e1a5f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682657860504 2023-04-28 04:57:42,932 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f44f3e4073244c31bc6a517504e88d37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682657860571 2023-04-28 04:57:42,932 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 38182ce692934f2c8984db3b9cc85a5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682657861694 2023-04-28 04:57:42,943 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:42,960 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/b51dde8a5be840148a380a050f24082d 2023-04-28 04:57:43,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:43,193 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:43,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657923214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657923214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657923215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657923215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657923225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657923326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657923326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657923326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657923326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657923327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,372 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/15b05d1f5c3a492a87330adf7706bca1 2023-04-28 04:57:43,375 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/d01ee5c77ba74127be727e7ecd0f211a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/d01ee5c77ba74127be727e7ecd0f211a 2023-04-28 04:57:43,379 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into d01ee5c77ba74127be727e7ecd0f211a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:43,379 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:43,379 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=13, startTime=1682657862888; duration=0sec 2023-04-28 04:57:43,379 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:43,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657923531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657923532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657923532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657923532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657923532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,787 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/612853b2f67a4eb088d36ac51e57cd72 2023-04-28 04:57:43,790 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/b51dde8a5be840148a380a050f24082d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b51dde8a5be840148a380a050f24082d 2023-04-28 04:57:43,793 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b51dde8a5be840148a380a050f24082d, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 04:57:43,794 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/15b05d1f5c3a492a87330adf7706bca1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/15b05d1f5c3a492a87330adf7706bca1 2023-04-28 04:57:43,797 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/15b05d1f5c3a492a87330adf7706bca1, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 04:57:43,797 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/612853b2f67a4eb088d36ac51e57cd72 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/612853b2f67a4eb088d36ac51e57cd72 2023-04-28 04:57:43,800 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/612853b2f67a4eb088d36ac51e57cd72, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 04:57:43,801 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 51e65c0b85e192dace926daeb81e88b4 in 871ms, sequenceid=95, compaction requested=false 2023-04-28 04:57:43,801 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:43,801 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:43,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:43,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:43,801 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:43,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:43,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,810 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:43,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,811 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:43,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:43,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:43,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79c2436b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:43,812 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:43,822 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:43,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:43,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:43,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:43,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:43,832 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:43,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:43,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:43,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:43,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:43,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:43,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:43,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:43,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 04:57:43,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:43,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:43,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:43,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:43,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:43,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:43,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:43,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,842 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:43,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:43,847 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,863 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:43,863 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:43,863 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:43,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:43,864 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:43,864 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:43,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:43,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:43,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657863863, max timestamp=1682657863873Num uniques 859; 2023-04-28 04:57:43,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:57:43,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657863863, max timestamp=1682657863873Num uniques 859; 2023-04-28 04:57:43,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:57:43,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657863863, max timestamp=1682657863873Num uniques 859; 2023-04-28 04:57:43,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:57:43,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657923907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657923908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657923925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657923925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:43,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657923926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,964 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:43,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:43,966 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:43,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:43,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:43,967 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:43,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:43,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:43,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:43,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,967 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:43,998 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:43,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:43,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:43,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:43,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:43,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:43,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:43,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:43,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:43,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:43,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:43,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:43,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:43,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:43,999 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:44,000 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:44,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:44,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:44,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:44,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,010 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ede9906[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:44,013 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:44,013 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,021 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:44,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:44,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:44,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:44,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,033 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:44,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,034 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:44,034 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657924026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657924026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:44,038 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:44,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:44,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:44,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:44,038 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:44,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657924039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657924039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657924039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,053 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,055 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,056 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,057 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,063 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:44,063 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:44,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,063 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,064 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:44,064 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:44,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,164 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:44,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:44,166 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:44,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:44,167 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:44,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:44,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:44,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:44,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:44,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:44,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:44,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:44,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:44,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:44,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:44,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:44,181 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:44,181 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:44,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:44,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:44,181 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:44,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,190 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:44,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740125bd[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:44,191 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:44,191 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:44,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:44,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:44,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:44,210 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,210 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,210 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,210 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:44,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:44,212 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:44,212 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:44,212 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:44,212 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:44,212 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:44,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,221 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,222 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657924235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657924235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657924243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657924244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657924245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/19cac5128a52447780682ef53931feeb 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,262 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,262 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,263 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:44,263 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:44,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:44,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/034b3a85cfa94f0083fc0f40ca914629 2023-04-28 04:57:44,363 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:44,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:44,365 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:44,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:44,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:44,366 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:44,366 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,366 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:44,366 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:44,366 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,366 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,377 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:44,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:44,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:44,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:44,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:44,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:44,378 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:44,378 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,378 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:44,378 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:44,378 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:44,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:44,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:44,379 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:44,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,389 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:44,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:44,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64a631c9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:44,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:44,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:44,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:44,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:44,410 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:44,410 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,410 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,410 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:44,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:44,412 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:44,412 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:44,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:44,413 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:44,413 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:44,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,421 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,431 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:44,431 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:44,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,431 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,432 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,432 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,432 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,532 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:44,534 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:44,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:44,535 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:44,535 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,535 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:44,535 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:44,536 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,536 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657924542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657924542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,547 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:44,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:44,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:44,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:44,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:44,549 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:44,549 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:44,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657924548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657924548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657924549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:44,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:44,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f14ef2d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:44,560 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,568 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,568 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:44,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:44,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:44,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:44,579 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:44,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:44,583 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:44,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:44,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:44,583 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:44,583 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:44,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:44,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:44,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,591 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,600 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:44,600 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:44,600 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:44,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,600 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:44,600 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:44,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:44,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,604 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:44,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:44,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/c818f9af430a4af3a5f2f50ecf157bf3 2023-04-28 04:57:44,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/19cac5128a52447780682ef53931feeb as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/19cac5128a52447780682ef53931feeb 2023-04-28 04:57:44,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/19cac5128a52447780682ef53931feeb, entries=200, sequenceid=107, filesize=13.9 K 2023-04-28 04:57:44,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/034b3a85cfa94f0083fc0f40ca914629 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/034b3a85cfa94f0083fc0f40ca914629 2023-04-28 04:57:44,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/034b3a85cfa94f0083fc0f40ca914629, entries=150, sequenceid=107, filesize=11.6 K 2023-04-28 04:57:44,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/c818f9af430a4af3a5f2f50ecf157bf3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c818f9af430a4af3a5f2f50ecf157bf3 2023-04-28 04:57:44,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c818f9af430a4af3a5f2f50ecf157bf3, entries=150, sequenceid=107, filesize=11.6 K 2023-04-28 04:57:44,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 51e65c0b85e192dace926daeb81e88b4 in 850ms, sequenceid=107, compaction requested=true 2023-04-28 04:57:44,688 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:44,688 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:44,688 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:44,688 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:44,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:44,689 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:44,691 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:44,691 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:44,691 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:44,691 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:44,691 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,691 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,691 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0480db5177524d37bea328734f57f8fb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b51dde8a5be840148a380a050f24082d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/19cac5128a52447780682ef53931feeb] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=37.2 K 2023-04-28 04:57:44,691 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/20e4a1f56c3c420594c077b1f1d5fab4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/15b05d1f5c3a492a87330adf7706bca1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/034b3a85cfa94f0083fc0f40ca914629] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=34.9 K 2023-04-28 04:57:44,691 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0480db5177524d37bea328734f57f8fb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682657861694 2023-04-28 04:57:44,691 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 20e4a1f56c3c420594c077b1f1d5fab4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682657861694 2023-04-28 04:57:44,692 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b51dde8a5be840148a380a050f24082d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682657862054 2023-04-28 04:57:44,692 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 15b05d1f5c3a492a87330adf7706bca1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682657862054 2023-04-28 04:57:44,692 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 19cac5128a52447780682ef53931feeb, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682657863192 2023-04-28 04:57:44,692 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 034b3a85cfa94f0083fc0f40ca914629, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682657863192 2023-04-28 04:57:44,696 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:44,697 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:44,701 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:44,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:44,703 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:44,705 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:44,705 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,705 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:44,705 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:44,706 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,706 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,714 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/5d921e8fedd24fc3b1f81deba76d9750 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5d921e8fedd24fc3b1f81deba76d9750 2023-04-28 04:57:44,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:44,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:44,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:44,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:44,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:44,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:44,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:44,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:44,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:44,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:44,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:44,717 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:44,717 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,717 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:44,717 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-28 04:57:44,718 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into 5d921e8fedd24fc3b1f81deba76d9750(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:44,718 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:44,718 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:44,718 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:44,718 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=13, startTime=1682657864688; duration=0sec 2023-04-28 04:57:44,718 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:44,718 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:44,718 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:44,718 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:44,718 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:44,718 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:44,719 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:44,719 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:44,719 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:44,719 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/d01ee5c77ba74127be727e7ecd0f211a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/612853b2f67a4eb088d36ac51e57cd72, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c818f9af430a4af3a5f2f50ecf157bf3] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=34.9 K 2023-04-28 04:57:44,720 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d01ee5c77ba74127be727e7ecd0f211a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682657861694 2023-04-28 04:57:44,721 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 612853b2f67a4eb088d36ac51e57cd72, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682657862054 2023-04-28 04:57:44,721 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c818f9af430a4af3a5f2f50ecf157bf3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682657863192 2023-04-28 04:57:44,729 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#356 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:45,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:45,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:45,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657925058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657925058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657925063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657925063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657925063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,114 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/0def8771bc294dceacbb683f37b40c63 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0def8771bc294dceacbb683f37b40c63 2023-04-28 04:57:45,117 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into 0def8771bc294dceacbb683f37b40c63(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:45,117 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:45,117 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=13, startTime=1682657864688; duration=0sec 2023-04-28 04:57:45,117 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:45,136 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/b347cdc193a84d6d8a8edbdca0bd5ba8 2023-04-28 04:57:45,147 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/bcdbf3f2ed394ff09e323f7dc9ebd025 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/bcdbf3f2ed394ff09e323f7dc9ebd025 2023-04-28 04:57:45,173 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into bcdbf3f2ed394ff09e323f7dc9ebd025(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:45,173 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:45,173 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=13, startTime=1682657864689; duration=0sec 2023-04-28 04:57:45,173 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657925165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657925169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657925169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657925170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657925375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657925375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657925376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657925376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,580 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/797250f6e6214564b39263cec59ffcd7 2023-04-28 04:57:45,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657925680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657925680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657925681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657925682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:45,988 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/22e0f42ac023423bab8507b28d9c2205 2023-04-28 04:57:45,992 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/b347cdc193a84d6d8a8edbdca0bd5ba8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b347cdc193a84d6d8a8edbdca0bd5ba8 2023-04-28 04:57:45,998 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b347cdc193a84d6d8a8edbdca0bd5ba8, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 04:57:45,998 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/797250f6e6214564b39263cec59ffcd7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/797250f6e6214564b39263cec59ffcd7 2023-04-28 04:57:46,002 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/797250f6e6214564b39263cec59ffcd7, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 04:57:46,002 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/22e0f42ac023423bab8507b28d9c2205 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/22e0f42ac023423bab8507b28d9c2205 2023-04-28 04:57:46,005 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/22e0f42ac023423bab8507b28d9c2205, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 04:57:46,006 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 51e65c0b85e192dace926daeb81e88b4 in 1289ms, sequenceid=144, compaction requested=false 2023-04-28 04:57:46,006 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:46,006 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:46,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:46,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:46,006 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:46,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:46,084 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,084 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,084 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,084 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:46,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45484160[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:46,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:46,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:46,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:46,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:46,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:46,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:46,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:46,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:46,107 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:46,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:46,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:46,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:46,107 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:46,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,117 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:46,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,126 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:46,126 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:46,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:46,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657866127, max timestamp=1682657866167Num uniques 859; 2023-04-28 04:57:46,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657866127, max timestamp=1682657866167Num uniques 859; 2023-04-28 04:57:46,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:57:46,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657866127, max timestamp=1682657866167Num uniques 859; 2023-04-28 04:57:46,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:57:46,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:57:46,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657926207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657926209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657926209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657926210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657926210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,227 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:46,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:46,228 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:46,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:46,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:46,229 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:46,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:46,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:46,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:46,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:46,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:46,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:46,276 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,276 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:46,276 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:46,276 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:46,276 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:46,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,295 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,295 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:46,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3819b8b9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:46,296 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:46,296 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,305 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:46,305 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:46,305 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:46,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,305 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:46,315 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:46,315 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:46,316 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:46,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:46,318 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:46,318 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:46,318 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:46,318 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:46,318 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:46,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657926314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657926315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657926317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657926317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657926317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,327 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,336 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,337 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:46,337 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,337 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,337 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,337 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,437 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:46,439 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:46,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:46,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:46,441 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:46,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:46,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:46,442 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,442 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:46,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:46,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:46,454 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,454 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:46,454 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:46,454 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:46,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:46,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:46,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de4fe0a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:46,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:46,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:46,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:46,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:46,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:46,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/b414a70730fa4a9aa712ec1db7a927f7 2023-04-28 04:57:46,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:46,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:46,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:46,486 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:46,486 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:46,486 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:46,486 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:46,486 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:46,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,505 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,505 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:46,505 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,505 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:46,506 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:46,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,506 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657926521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657926522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657926523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657926523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657926523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,606 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:46,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:46,607 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:46,608 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:46,608 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,608 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:46,608 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:46,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,621 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,624 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:46,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:46,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:46,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:46,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:46,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:46,627 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:46,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:46,628 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,628 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:46,628 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:46,628 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:46,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:46,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:46,628 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:46,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a1191c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:46,642 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:46,642 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,652 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:46,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:46,653 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:46,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:46,663 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:46,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:46,666 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:46,666 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:46,666 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:46,666 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:46,666 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:46,673 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,673 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,673 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,675 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,684 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:46,684 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,684 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,684 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:46,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:46,685 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:46,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,685 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,785 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:46,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:46,788 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:46,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:46,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:46,790 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:46,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:46,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:46,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:46,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:46,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:46,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:46,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:46,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:46,804 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:46,804 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,805 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:46,805 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:46,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657926827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657926827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657926828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657926829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:46,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657926831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,844 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,844 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,845 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,845 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:46,846 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:46,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42651389[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:46,846 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,863 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:46,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:46,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:46,863 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,863 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:46,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,873 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:46,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:46,875 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:46,876 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:46,876 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:46,876 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:46,876 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:46,884 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,884 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:46,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,884 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:46,885 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,886 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/a92fabb7c87e4fe8b68af46715b5fea9 2023-04-28 04:57:46,894 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,894 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:46,895 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:46,895 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:46,895 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,895 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:46,895 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:46,895 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:46,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:46,995 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:46,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:46,997 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:46,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:46,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:46,998 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:46,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:46,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:46,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:46,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:46,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,010 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,011 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,011 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:47,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:47,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:47,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:47,011 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:47,012 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,012 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:47,012 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:47,012 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:47,012 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:47,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,021 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,021 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:47,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:47,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37cca70c[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:47,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,031 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,031 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:47,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:47,031 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:47,032 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:47,042 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:47,042 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,042 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,042 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:47,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:47,044 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:47,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:47,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:47,044 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:47,044 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:47,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,053 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,063 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,063 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:47,064 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:47,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,064 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,064 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:47,064 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:47,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,164 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:47,166 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:47,167 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:47,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:47,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:47,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,179 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:47,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:47,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:47,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:47,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:47,180 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:47,180 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:47,189 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:47,190 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:47,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d920b51[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:47,191 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:47,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:47,200 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:47,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:47,210 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:47,210 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,210 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,210 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:47,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:47,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:47,215 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:47,215 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:47,215 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:47,215 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:47,215 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:47,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,221 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,224 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,231 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,231 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,233 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,233 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:47,233 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,233 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,233 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:47,234 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:47,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:47,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/849d761235744ae1a97988642171fa07 2023-04-28 04:57:47,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/b414a70730fa4a9aa712ec1db7a927f7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b414a70730fa4a9aa712ec1db7a927f7 2023-04-28 04:57:47,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b414a70730fa4a9aa712ec1db7a927f7, entries=200, sequenceid=156, filesize=14.1 K 2023-04-28 04:57:47,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/a92fabb7c87e4fe8b68af46715b5fea9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a92fabb7c87e4fe8b68af46715b5fea9 2023-04-28 04:57:47,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a92fabb7c87e4fe8b68af46715b5fea9, entries=150, sequenceid=156, filesize=11.8 K 2023-04-28 04:57:47,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/849d761235744ae1a97988642171fa07 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/849d761235744ae1a97988642171fa07 2023-04-28 04:57:47,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/849d761235744ae1a97988642171fa07, entries=150, sequenceid=156, filesize=11.8 K 2023-04-28 04:57:47,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 51e65c0b85e192dace926daeb81e88b4 in 1246ms, sequenceid=156, compaction requested=true 2023-04-28 04:57:47,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:47,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:47,320 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:47,320 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:47,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:47,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:47,321 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38554 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:47,321 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:47,321 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,321 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0def8771bc294dceacbb683f37b40c63, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b347cdc193a84d6d8a8edbdca0bd5ba8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b414a70730fa4a9aa712ec1db7a927f7] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=37.7 K 2023-04-28 04:57:47,321 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:47,321 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:47,321 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,321 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5d921e8fedd24fc3b1f81deba76d9750, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/797250f6e6214564b39263cec59ffcd7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a92fabb7c87e4fe8b68af46715b5fea9] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=35.3 K 2023-04-28 04:57:47,321 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 0def8771bc294dceacbb683f37b40c63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682657863192 2023-04-28 04:57:47,324 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5d921e8fedd24fc3b1f81deba76d9750, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682657863192 2023-04-28 04:57:47,331 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b347cdc193a84d6d8a8edbdca0bd5ba8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682657863907 2023-04-28 04:57:47,331 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 797250f6e6214564b39263cec59ffcd7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682657863907 2023-04-28 04:57:47,333 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a92fabb7c87e4fe8b68af46715b5fea9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682657865056 2023-04-28 04:57:47,333 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b414a70730fa4a9aa712ec1db7a927f7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682657865056 2023-04-28 04:57:47,334 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:47,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:47,336 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:47,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:47,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-28 04:57:47,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:47,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:57:47,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:47,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:57:47,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:47,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:57:47,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:47,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:47,348 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:47,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:47,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:47,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,349 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#362 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:47,353 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:47,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657927343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657927343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657927345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657927346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657927348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,358 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:47,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:47,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:47,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:47,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:47,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:47,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:47,373 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:47,373 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:47,373 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:47,373 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:47,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:47,373 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:47,380 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:47,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,389 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:47,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b68406[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:47,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:47,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:47,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:47,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:47,435 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:47,435 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,436 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,436 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:47,437 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:47,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:47,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:47,437 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:47,438 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:47,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,454 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657927455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657927455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,463 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:47,463 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:47,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,464 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,464 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,464 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,464 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,464 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:47,464 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:47,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:47,466 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,466 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657927460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657927460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657927460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,564 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:47,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:47,566 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:47,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:47,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:47,567 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:47,567 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,567 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:47,567 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:47,567 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,567 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,579 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:47,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:47,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:47,584 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,584 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:47,584 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:47,584 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:47,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:47,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,590 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,590 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aa530c2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:47,591 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:47,591 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:47,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:47,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:47,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:47,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:47,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:47,612 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:47,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:47,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:47,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:47,612 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:47,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,621 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,621 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,622 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,631 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,631 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,631 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,631 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,631 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:47,631 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:47,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,631 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:47,631 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,632 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,632 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:47,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,632 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657927662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657927662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657927668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657927668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657927669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,732 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:47,734 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:47,746 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:47,746 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,746 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:47,746 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:47,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,758 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,758 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:47,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:47,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:47,763 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,763 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:47,763 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:47,763 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:47,763 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:47,764 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,764 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:47,764 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:47,768 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:47,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@883085d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:47,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:47,770 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,779 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:47,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:47,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:47,779 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,779 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:47,781 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/6e75bc4cd0414f448b30714b848284b9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/6e75bc4cd0414f448b30714b848284b9 2023-04-28 04:57:47,785 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into 6e75bc4cd0414f448b30714b848284b9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:47,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/4fb08ba38faf40f6b262e47ac790305e 2023-04-28 04:57:47,785 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:47,785 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=13, startTime=1682657867320; duration=0sec 2023-04-28 04:57:47,785 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:47,785 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:47,786 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:47,786 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:47,786 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,786 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/2f9205ef3d89464698acc8340f9bfca6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/2f9205ef3d89464698acc8340f9bfca6 2023-04-28 04:57:47,786 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/bcdbf3f2ed394ff09e323f7dc9ebd025, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/22e0f42ac023423bab8507b28d9c2205, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/849d761235744ae1a97988642171fa07] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=35.3 K 2023-04-28 04:57:47,786 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting bcdbf3f2ed394ff09e323f7dc9ebd025, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682657863192 2023-04-28 04:57:47,787 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 22e0f42ac023423bab8507b28d9c2205, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682657863907 2023-04-28 04:57:47,787 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 849d761235744ae1a97988642171fa07, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682657865056 2023-04-28 04:57:47,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:47,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,790 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,790 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into 2f9205ef3d89464698acc8340f9bfca6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:47,790 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,790 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=13, startTime=1682657867320; duration=0sec 2023-04-28 04:57:47,790 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:47,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:47,791 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:47,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:47,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:47,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:47,791 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:47,792 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#365 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:47,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,801 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,810 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,810 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:47,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,811 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,810 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:47,811 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:47,811 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:47,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:47,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:47,911 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:47,913 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:47,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:47,914 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:47,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:47,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:47,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:47,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:47,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:47,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:47,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:47,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:47,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:47,935 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,935 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:47,935 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:47,935 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:47,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:47,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:47,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:47,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6526529f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:47,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:47,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:47,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:47,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:47,958 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:47,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,968 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:47,968 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:47,968 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:47,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:47,970 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:47,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:47,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:47,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:47,970 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:47,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657927968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657927969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657927974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657927974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:47,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:47,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657927975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,037 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:48,037 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,038 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,089 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:48,089 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,090 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:48,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,090 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:48,090 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:48,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:48,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:48,090 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:48,089 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:48,090 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:48,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:48,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,090 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,190 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:48,192 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:48,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:48,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:48,193 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:48,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:48,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:48,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:48,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,193 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/3afed8d39f5d4478afc315235a952e67 2023-04-28 04:57:48,210 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:48,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:48,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:48,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:48,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:48,211 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:48,211 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:48,211 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:48,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:48,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:48,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:48,212 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:48,212 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:48,220 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/649bf87931fa437d9b9589787d05014e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/649bf87931fa437d9b9589787d05014e 2023-04-28 04:57:48,221 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,221 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,221 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:48,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b46c89[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:48,223 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:48,223 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,224 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into 649bf87931fa437d9b9589787d05014e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:48,224 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:48,224 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=13, startTime=1682657867320; duration=0sec 2023-04-28 04:57:48,224 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:48,231 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,231 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:48,231 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:48,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:48,232 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,232 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:48,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:48,242 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:48,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:48,244 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:48,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:48,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:48,244 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:48,244 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:48,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,253 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:48,322 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:48,322 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:48,322 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,322 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:48,322 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:48,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:48,322 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:48,423 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:48,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:48,424 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:48,425 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:48,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:48,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:48,425 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:48,426 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,426 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,431 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:48,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:48,431 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,431 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:48,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:48,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:48,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:48,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:48,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:48,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:48,433 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:48,433 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:48,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:48,433 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:48,433 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:48,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:48,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:48,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:48,433 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:48,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12e4735d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:48,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:48,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,452 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,452 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:48,452 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:48,453 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:48,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,453 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:48,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:48,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:48,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:48,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:48,465 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:48,465 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:48,465 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:48,465 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:48,465 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:48,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:48,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,475 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:48,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657928472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657928479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657928480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657928481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:48,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657928481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:48,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:48,484 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:48,484 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:48,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,484 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,484 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:48,484 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,585 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:48,586 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:48,587 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:48,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:48,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:48,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:48,588 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,588 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/a6f36e74b4fe452283885be65606f255 2023-04-28 04:57:48,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/4fb08ba38faf40f6b262e47ac790305e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/4fb08ba38faf40f6b262e47ac790305e 2023-04-28 04:57:48,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/4fb08ba38faf40f6b262e47ac790305e, entries=250, sequenceid=195, filesize=16.4 K 2023-04-28 04:57:48,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/3afed8d39f5d4478afc315235a952e67 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/3afed8d39f5d4478afc315235a952e67 2023-04-28 04:57:48,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/3afed8d39f5d4478afc315235a952e67, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 04:57:48,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/a6f36e74b4fe452283885be65606f255 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/a6f36e74b4fe452283885be65606f255 2023-04-28 04:57:48,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/a6f36e74b4fe452283885be65606f255, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 04:57:48,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=33.54 KB/34350 for 51e65c0b85e192dace926daeb81e88b4 in 1296ms, sequenceid=195, compaction requested=false 2023-04-28 04:57:48,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:48,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,654 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:48,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:48,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:48,654 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:48,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:48,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:48,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:48,655 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:48,655 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:48,655 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:48,655 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:48,656 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:48,656 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:48,656 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:49,061 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/2bf6fac20b1e4f628edc1a224ca6b4bc 2023-04-28 04:57:49,468 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/90f7da913bb04203bff357bee6006566 2023-04-28 04:57:49,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:49,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:49,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657869499, max timestamp=1682657869510Num uniques 859; 2023-04-28 04:57:49,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:57:49,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657869499, max timestamp=1682657869510Num uniques 859; 2023-04-28 04:57:49,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:57:49,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657869499, max timestamp=1682657869510Num uniques 859; 2023-04-28 04:57:49,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:57:49,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657929542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657929545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657929546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657929547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657929547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657929650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657929650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657929654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657929654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57162 deadline: 1682657929656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657929856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657929857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657929857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657929863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:49,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657929858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,875 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/772cf5572ae240f7b5a814dca9f93de7 2023-04-28 04:57:49,880 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/2bf6fac20b1e4f628edc1a224ca6b4bc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/2bf6fac20b1e4f628edc1a224ca6b4bc 2023-04-28 04:57:49,886 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/2bf6fac20b1e4f628edc1a224ca6b4bc, entries=150, sequenceid=206, filesize=11.8 K 2023-04-28 04:57:49,887 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/90f7da913bb04203bff357bee6006566 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/90f7da913bb04203bff357bee6006566 2023-04-28 04:57:49,891 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/90f7da913bb04203bff357bee6006566, entries=150, sequenceid=206, filesize=11.8 K 2023-04-28 04:57:49,891 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/772cf5572ae240f7b5a814dca9f93de7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/772cf5572ae240f7b5a814dca9f93de7 2023-04-28 04:57:49,896 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/772cf5572ae240f7b5a814dca9f93de7, entries=150, sequenceid=206, filesize=11.8 K 2023-04-28 04:57:49,897 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 51e65c0b85e192dace926daeb81e88b4 in 1241ms, sequenceid=206, compaction requested=true 2023-04-28 04:57:49,897 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:49,897 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:49,897 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:49,897 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:49,897 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,898 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:49,898 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:49,905 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:49,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:49,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b540bc[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:49,907 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:49,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:49,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:49,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:49,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:49,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:49,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:49,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:49,926 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:49,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:49,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:49,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:49,926 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:49,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:49,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,937 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:49,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:49,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:49,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,939 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:49,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:49,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:49,948 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:49,948 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:49,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:49,948 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:49,948 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:49,948 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:49,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:50,048 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:50,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:50,050 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:50,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:50,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:50,051 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:50,051 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:50,051 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:50,051 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:50,052 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,052 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:50,063 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:50,063 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:50,063 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:50,063 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:50,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:50,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:50,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:50,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:50,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:50,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:50,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:50,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:50,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:50,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:50,065 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:50,065 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:50,065 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-28 04:57:50,066 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:50,066 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:50,066 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:50,066 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:50,066 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:50,066 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:50,077 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/166d0621789a4634ba56c3bd0a118376 2023-04-28 04:57:50,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:50,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:50,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657930173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657930174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657930175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657930176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657930177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657930284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657930284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657930284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657930284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657930284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,488 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/e475fa287d384434b670584d9f076472 2023-04-28 04:57:50,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657930493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657930494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657930494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657930495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657930495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657930797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657930797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657930799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657930801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:50,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657930801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,896 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/7e7ff484a81c411987a62a45557c8929 2023-04-28 04:57:50,899 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/166d0621789a4634ba56c3bd0a118376 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/166d0621789a4634ba56c3bd0a118376 2023-04-28 04:57:50,902 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/166d0621789a4634ba56c3bd0a118376, entries=150, sequenceid=242, filesize=11.8 K 2023-04-28 04:57:50,902 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/e475fa287d384434b670584d9f076472 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/e475fa287d384434b670584d9f076472 2023-04-28 04:57:50,905 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/e475fa287d384434b670584d9f076472, entries=150, sequenceid=242, filesize=11.8 K 2023-04-28 04:57:50,905 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/7e7ff484a81c411987a62a45557c8929 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/7e7ff484a81c411987a62a45557c8929 2023-04-28 04:57:50,907 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/7e7ff484a81c411987a62a45557c8929, entries=150, sequenceid=242, filesize=11.8 K 2023-04-28 04:57:50,908 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for 51e65c0b85e192dace926daeb81e88b4 in 843ms, sequenceid=242, compaction requested=true 2023-04-28 04:57:50,908 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:50,908 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:50,908 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:50,908 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:50,908 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,908 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:50,908 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:50,926 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:50,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:50,926 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:50,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:50,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d058df[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:50,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:50,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,936 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:50,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:50,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:50,937 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:50,947 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:50,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:50,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:50,947 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:50,949 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:50,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:50,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:50,949 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:50,949 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:50,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:50,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:50,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:50,959 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:50,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:50,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:50,968 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:50,968 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:50,968 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:50,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:50,968 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:50,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:50,968 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:50,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:50,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:50,969 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:51,069 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:51,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:51,070 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:51,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:51,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:51,071 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:51,071 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:51,072 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:51,072 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:51,072 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:51,072 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:51,084 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:51,084 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:51,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:51,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,092 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:51,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:51,092 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:51,092 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:51,092 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:51,092 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:51,092 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:51,092 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:51,093 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:51,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:51,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:51,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:51,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657871322, max timestamp=1682657871332Num uniques 859; 2023-04-28 04:57:51,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:57:51,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657871322, max timestamp=1682657871332Num uniques 859; 2023-04-28 04:57:51,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:57:51,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682657871322, max timestamp=1682657871332Num uniques 859; 2023-04-28 04:57:51,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-28 04:57:51,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657931357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657931361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657931367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657931368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657931368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657931469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657931469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657931469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657931476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657931476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,505 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/7e77f33783f443359396855b616b2144 2023-04-28 04:57:51,516 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/91595bf705264f339d7ab2f77169e4a6 2023-04-28 04:57:51,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57254 deadline: 1682657931676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57196 deadline: 1682657931676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657931677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57246 deadline: 1682657931684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:51,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:57220 deadline: 1682657931688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,925 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/cd6ceee68e0a4c4b9dd41ab7b56063f6 2023-04-28 04:57:51,929 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/7e77f33783f443359396855b616b2144 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7e77f33783f443359396855b616b2144 2023-04-28 04:57:51,933 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7e77f33783f443359396855b616b2144, entries=150, sequenceid=250, filesize=11.8 K 2023-04-28 04:57:51,934 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/91595bf705264f339d7ab2f77169e4a6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/91595bf705264f339d7ab2f77169e4a6 2023-04-28 04:57:51,944 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/91595bf705264f339d7ab2f77169e4a6, entries=150, sequenceid=250, filesize=11.8 K 2023-04-28 04:57:51,945 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/cd6ceee68e0a4c4b9dd41ab7b56063f6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/cd6ceee68e0a4c4b9dd41ab7b56063f6 2023-04-28 04:57:51,948 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/cd6ceee68e0a4c4b9dd41ab7b56063f6, entries=150, sequenceid=250, filesize=11.8 K 2023-04-28 04:57:51,949 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=161.02 KB/164880 for 51e65c0b85e192dace926daeb81e88b4 in 856ms, sequenceid=250, compaction requested=true 2023-04-28 04:57:51,949 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:51,949 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:51,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:51,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:51,949 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,952 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:51,952 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:51,968 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,968 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:51,968 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:51,968 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:51,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:51,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1742b6c8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:51,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:51,970 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:51,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:51,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:51,979 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:51,979 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:51,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,979 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:51,979 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:51,989 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:51,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:51,989 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:51,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:51,992 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:51,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:51,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:51,992 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:51,992 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:51,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:51,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657871360, max timestamp=1682657871994Num uniques 1218; 2023-04-28 04:57:51,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-04-28 04:57:51,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=120.76 KB heapSize=332.83 KB 2023-04-28 04:57:51,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657871360, max timestamp=1682657871994Num uniques 1218; 2023-04-28 04:57:51,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682657871360, max timestamp=1682657871994Num uniques 1218; 2023-04-28 04:57:51,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:51,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-04-28 04:57:51,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:51,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-04-28 04:57:51,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:51,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:51,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:51,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:52,000 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,000 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,000 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,001 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,004 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,011 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,010 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:52,014 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,014 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:52,014 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:52,014 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:52,014 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657872019, max timestamp=1682657872044Num uniques 859; 2023-04-28 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657872019, max timestamp=1682657872044Num uniques 859; 2023-04-28 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:52,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657872019, max timestamp=1682657872044Num uniques 859; 2023-04-28 04:57:52,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:57:52,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682657872056, max timestamp=1682657872089Num uniques 1068; 2023-04-28 04:57:52,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-28 04:57:52,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682657872056, max timestamp=1682657872089Num uniques 1068; 2023-04-28 04:57:52,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-28 04:57:52,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682657872056, max timestamp=1682657872089Num uniques 1068; 2023-04-28 04:57:52,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-28 04:57:52,114 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:52,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:52,116 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:52,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:52,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:52,118 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:52,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:52,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:52,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,118 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:52,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:52,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:52,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:52,127 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:52,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:52,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:52,128 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:52,128 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:52,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657932122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,137 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:52,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657932133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1682657872117, max timestamp=1682657872135Num uniques 1227; 2023-04-28 04:57:52,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=14972, suffix heap size=155576,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=327 2023-04-28 04:57:52,139 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1682657872117, max timestamp=1682657872135Num uniques 1127; 2023-04-28 04:57:52,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-28 04:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1682657872117, max timestamp=1682657872135Num uniques 1227; 2023-04-28 04:57:52,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=14972, suffix heap size=155576,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=327 2023-04-28 04:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:52,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@268d41e6[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:52,141 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:52,141 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,147 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:52,147 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:52,147 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:52,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,148 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:52,157 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:52,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:52,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:52,160 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:52,160 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:52,160 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:52,160 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:52,160 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:52,168 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,168 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,168 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,169 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,170 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657932194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657872174, max timestamp=1682657872203Num uniques 1186; 2023-04-28 04:57:52,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-28 04:57:52,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657872174, max timestamp=1682657872203Num uniques 1186; 2023-04-28 04:57:52,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-28 04:57:52,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657872174, max timestamp=1682657872203Num uniques 1186; 2023-04-28 04:57:52,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-28 04:57:52,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,211 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:52,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:52,211 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:52,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,212 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:52,211 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,212 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:52,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,212 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657872267, max timestamp=1682657872286Num uniques 1095; 2023-04-28 04:57:52,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 04:57:52,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657872267, max timestamp=1682657872286Num uniques 1095; 2023-04-28 04:57:52,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 04:57:52,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657872267, max timestamp=1682657872286Num uniques 1095; 2023-04-28 04:57:52,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-28 04:57:52,312 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:52,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:52,314 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:52,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:52,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:52,314 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:52,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:52,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:52,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682657872313, max timestamp=1682657872332Num uniques 1054; 2023-04-28 04:57:52,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-28 04:57:52,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682657872313, max timestamp=1682657872332Num uniques 1054; 2023-04-28 04:57:52,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682657872313, max timestamp=1682657872332Num uniques 1054; 2023-04-28 04:57:52,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-28 04:57:52,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-28 04:57:52,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,336 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:52,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:52,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:52,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,337 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:52,338 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,338 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:52,338 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:52,338 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:52,338 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:52,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:52,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e95552b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:52,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:52,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,357 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,358 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:52,358 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:52,358 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:52,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:52,368 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:52,368 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,368 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,368 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657872366, max timestamp=1682657872369Num uniques 1113; 2023-04-28 04:57:52,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-28 04:57:52,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657872366, max timestamp=1682657872369Num uniques 1113; 2023-04-28 04:57:52,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-28 04:57:52,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657872366, max timestamp=1682657872369Num uniques 1113; 2023-04-28 04:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-28 04:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:52,373 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:52,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:52,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:52,373 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:52,373 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:52,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,389 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:52,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:52,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,400 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:52,400 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/01cb7211ad6046f2b079088f9b1e5a17 2023-04-28 04:57:52,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/41dae963b188461090afb28cf197f7b3 2023-04-28 04:57:52,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682657872401, max timestamp=1682657872431Num uniques 1022; 2023-04-28 04:57:52,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682657872401, max timestamp=1682657872431Num uniques 1022; 2023-04-28 04:57:52,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-28 04:57:52,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-28 04:57:52,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/e0b189b6ec1c44f09d3668b3ef34b3c6 2023-04-28 04:57:52,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682657872401, max timestamp=1682657872431Num uniques 1022; 2023-04-28 04:57:52,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-28 04:57:52,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/01cb7211ad6046f2b079088f9b1e5a17 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/01cb7211ad6046f2b079088f9b1e5a17 2023-04-28 04:57:52,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/01cb7211ad6046f2b079088f9b1e5a17, entries=200, sequenceid=288, filesize=14.3 K 2023-04-28 04:57:52,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/41dae963b188461090afb28cf197f7b3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/41dae963b188461090afb28cf197f7b3 2023-04-28 04:57:52,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/41dae963b188461090afb28cf197f7b3, entries=150, sequenceid=288, filesize=11.9 K 2023-04-28 04:57:52,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/e0b189b6ec1c44f09d3668b3ef34b3c6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/e0b189b6ec1c44f09d3668b3ef34b3c6 2023-04-28 04:57:52,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/e0b189b6ec1c44f09d3668b3ef34b3c6, entries=150, sequenceid=288, filesize=11.9 K 2023-04-28 04:57:52,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=80.51 KB/82440 for 51e65c0b85e192dace926daeb81e88b4 in 452ms, sequenceid=288, compaction requested=true 2023-04-28 04:57:52,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:52,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:52,449 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 04:57:52,449 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 04:57:52,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:52,450 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:52,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:52,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=87.22 KB heapSize=244.31 KB 2023-04-28 04:57:52,451 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72674 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 04:57:52,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:52,451 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:52,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:52,451 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:52,452 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/2f9205ef3d89464698acc8340f9bfca6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/3afed8d39f5d4478afc315235a952e67, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/90f7da913bb04203bff357bee6006566, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/e475fa287d384434b670584d9f076472, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/91595bf705264f339d7ab2f77169e4a6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/41dae963b188461090afb28cf197f7b3] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=71.0 K 2023-04-28 04:57:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:57:52,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 79894 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 04:57:52,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:52,452 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,452 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/6e75bc4cd0414f448b30714b848284b9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/4fb08ba38faf40f6b262e47ac790305e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/2bf6fac20b1e4f628edc1a224ca6b4bc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/166d0621789a4634ba56c3bd0a118376, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7e77f33783f443359396855b616b2144, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/01cb7211ad6046f2b079088f9b1e5a17] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=78.0 K 2023-04-28 04:57:52,452 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 2f9205ef3d89464698acc8340f9bfca6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682657865056 2023-04-28 04:57:52,452 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6e75bc4cd0414f448b30714b848284b9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682657865056 2023-04-28 04:57:52,453 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 3afed8d39f5d4478afc315235a952e67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682657866208 2023-04-28 04:57:52,453 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4fb08ba38faf40f6b262e47ac790305e, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682657866208 2023-04-28 04:57:52,454 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 2bf6fac20b1e4f628edc1a224ca6b4bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682657867343 2023-04-28 04:57:52,454 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 90f7da913bb04203bff357bee6006566, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682657867343 2023-04-28 04:57:52,454 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 166d0621789a4634ba56c3bd0a118376, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1682657869534 2023-04-28 04:57:52,454 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting e475fa287d384434b670584d9f076472, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1682657869534 2023-04-28 04:57:52,455 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7e77f33783f443359396855b616b2144, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682657870174 2023-04-28 04:57:52,455 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 91595bf705264f339d7ab2f77169e4a6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682657870174 2023-04-28 04:57:52,455 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 01cb7211ad6046f2b079088f9b1e5a17, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682657871360 2023-04-28 04:57:52,455 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 41dae963b188461090afb28cf197f7b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682657871360 2023-04-28 04:57:52,465 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#381 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:52,466 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#382 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:52,492 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/9310d8aa3cc349af96d0ef0788176e4a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/9310d8aa3cc349af96d0ef0788176e4a 2023-04-28 04:57:52,492 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/fc7d395280fe4fb584d5f3b7cc69f7e2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/fc7d395280fe4fb584d5f3b7cc69f7e2 2023-04-28 04:57:52,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657932482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,499 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into fc7d395280fe4fb584d5f3b7cc69f7e2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:52,499 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:52,499 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=10, startTime=1682657872449; duration=0sec 2023-04-28 04:57:52,499 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:52,499 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 04:57:52,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657932484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657932484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,501 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:52,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:52,504 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72674 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 04:57:52,504 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:52,504 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,505 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/649bf87931fa437d9b9589787d05014e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/a6f36e74b4fe452283885be65606f255, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/772cf5572ae240f7b5a814dca9f93de7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/7e7ff484a81c411987a62a45557c8929, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/cd6ceee68e0a4c4b9dd41ab7b56063f6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/e0b189b6ec1c44f09d3668b3ef34b3c6] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=71.0 K 2023-04-28 04:57:52,505 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 649bf87931fa437d9b9589787d05014e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1682657865056 2023-04-28 04:57:52,506 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting a6f36e74b4fe452283885be65606f255, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682657866208 2023-04-28 04:57:52,506 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:52,506 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 772cf5572ae240f7b5a814dca9f93de7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682657867343 2023-04-28 04:57:52,506 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7e7ff484a81c411987a62a45557c8929, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1682657869534 2023-04-28 04:57:52,506 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting cd6ceee68e0a4c4b9dd41ab7b56063f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682657870174 2023-04-28 04:57:52,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,507 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting e0b189b6ec1c44f09d3668b3ef34b3c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682657871360 2023-04-28 04:57:52,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657932486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657932489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,517 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into 9310d8aa3cc349af96d0ef0788176e4a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:52,517 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:52,517 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=10, startTime=1682657872449; duration=0sec 2023-04-28 04:57:52,518 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:52,530 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#383 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:52,532 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:52,533 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,533 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:52,533 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:52,533 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,533 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,541 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/60ac50343b7a4378b6efdca1dea88879 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/60ac50343b7a4378b6efdca1dea88879 2023-04-28 04:57:52,545 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,545 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,545 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,545 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:52,548 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into 60ac50343b7a4378b6efdca1dea88879(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:52,548 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:52,548 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=10, startTime=1682657872450; duration=0sec 2023-04-28 04:57:52,548 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:52,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:52,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:52,556 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:52,556 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,556 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:52,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:52,569 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:52,573 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,573 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:52,573 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:52,573 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:52,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:52,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:52,573 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:52,578 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:52,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59d4f286[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:52,580 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:52,580 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:52,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:52,589 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,589 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:52,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:52,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:52,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657932596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:52,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:52,602 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:52,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:52,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:52,602 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:52,602 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:52,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657932601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657932602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,612 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657932609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657932610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,622 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,622 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:52,622 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:52,622 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,623 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,623 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,623 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:52,623 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:52,723 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:52,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:52,725 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:52,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:52,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:52,740 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:52,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:52,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:52,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,740 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,747 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:52,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:52,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:52,749 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:52,749 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,749 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:52,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:52,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:52,752 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:52,752 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:52,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,758 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,758 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,758 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:52,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:52,759 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba2eb82[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:52,768 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,768 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:52,768 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:52,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:52,769 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:52,779 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:52,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:52,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:52,783 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:52,783 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:52,783 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:52,783 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:52,783 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:52,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:52,789 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:52,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,790 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,792 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,800 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,800 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:52,800 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:52,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,800 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:52,800 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,800 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657932802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657932806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657932807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657932815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:52,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657932815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/36aa3aa7a2ec48d6b770d04d693a1719 2023-04-28 04:57:52,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/582603a6344a400eba005a89b40ee235 2023-04-28 04:57:52,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/6ec1728020b546698de93181af256864 2023-04-28 04:57:52,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/36aa3aa7a2ec48d6b770d04d693a1719 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/36aa3aa7a2ec48d6b770d04d693a1719 2023-04-28 04:57:52,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/36aa3aa7a2ec48d6b770d04d693a1719, entries=200, sequenceid=438, filesize=14.3 K 2023-04-28 04:57:52,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/582603a6344a400eba005a89b40ee235 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/582603a6344a400eba005a89b40ee235 2023-04-28 04:57:52,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/582603a6344a400eba005a89b40ee235, entries=150, sequenceid=438, filesize=11.9 K 2023-04-28 04:57:52,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/6ec1728020b546698de93181af256864 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/6ec1728020b546698de93181af256864 2023-04-28 04:57:52,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/6ec1728020b546698de93181af256864, entries=150, sequenceid=438, filesize=11.9 K 2023-04-28 04:57:52,897 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~244.27 KB/250128, currentSize=93.93 KB/96180 for 51e65c0b85e192dace926daeb81e88b4 in 446ms, sequenceid=438, compaction requested=false 2023-04-28 04:57:52,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:52,900 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:52,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:52,909 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:52,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:52,910 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:52,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:52,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:52,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,910 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:52,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:52,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:52,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:52,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:52,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:52,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:52,918 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:52,918 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:52,918 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:52,918 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:52,918 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:52,918 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:52,918 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:52,918 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 04:57:52,919 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:52,919 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:52,919 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:52,919 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:52,919 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:52,919 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:53,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:53,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657933129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657933129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657933130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657933138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657933131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657933240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657933240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657933240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657933244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657933244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,329 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/f09965bdbeb14e3386bfa9389f8d74d6 2023-04-28 04:57:53,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657933445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657933446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657933446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657933450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657933451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,742 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/4e495c690f01415eb219f27c21b4b53d 2023-04-28 04:57:53,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657933750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657933754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657933756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657933757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:53,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:53,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657933759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,151 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/10bf7d43715643a2841d23ea8e305c48 2023-04-28 04:57:54,153 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/f09965bdbeb14e3386bfa9389f8d74d6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/f09965bdbeb14e3386bfa9389f8d74d6 2023-04-28 04:57:54,156 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/f09965bdbeb14e3386bfa9389f8d74d6, entries=150, sequenceid=458, filesize=11.9 K 2023-04-28 04:57:54,157 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/4e495c690f01415eb219f27c21b4b53d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/4e495c690f01415eb219f27c21b4b53d 2023-04-28 04:57:54,160 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/4e495c690f01415eb219f27c21b4b53d, entries=150, sequenceid=458, filesize=11.9 K 2023-04-28 04:57:54,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/10bf7d43715643a2841d23ea8e305c48 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/10bf7d43715643a2841d23ea8e305c48 2023-04-28 04:57:54,164 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/10bf7d43715643a2841d23ea8e305c48, entries=150, sequenceid=458, filesize=11.9 K 2023-04-28 04:57:54,165 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 51e65c0b85e192dace926daeb81e88b4 in 1247ms, sequenceid=458, compaction requested=true 2023-04-28 04:57:54,165 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:54,165 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:54,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:54,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:54,165 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:54,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,248 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,248 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,248 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:54,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176c1361[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:54,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:54,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:54,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 04:57:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:54,271 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:54,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:54,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:54,272 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,272 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:54,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/ac6dc14859b84fd6befd9f4719a42a0d 2023-04-28 04:57:54,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657934282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/5300d071889649109ce6f5a5b40143a7 2023-04-28 04:57:54,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657934283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657934284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657934285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657934285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/b9300b62d9934f7f9af83a0e8e0c4fcc 2023-04-28 04:57:54,304 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/ac6dc14859b84fd6befd9f4719a42a0d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/ac6dc14859b84fd6befd9f4719a42a0d 2023-04-28 04:57:54,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/ac6dc14859b84fd6befd9f4719a42a0d, entries=200, sequenceid=474, filesize=14.3 K 2023-04-28 04:57:54,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/5300d071889649109ce6f5a5b40143a7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5300d071889649109ce6f5a5b40143a7 2023-04-28 04:57:54,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5300d071889649109ce6f5a5b40143a7, entries=150, sequenceid=474, filesize=11.9 K 2023-04-28 04:57:54,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/b9300b62d9934f7f9af83a0e8e0c4fcc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b9300b62d9934f7f9af83a0e8e0c4fcc 2023-04-28 04:57:54,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b9300b62d9934f7f9af83a0e8e0c4fcc, entries=150, sequenceid=474, filesize=11.9 K 2023-04-28 04:57:54,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 51e65c0b85e192dace926daeb81e88b4 in 60ms, sequenceid=474, compaction requested=true 2023-04-28 04:57:54,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:54,321 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:54,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:54,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:54,321 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:54,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:54,322 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54136 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:54,322 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:54,322 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:54,322 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:54,322 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,322 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,322 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/fc7d395280fe4fb584d5f3b7cc69f7e2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/36aa3aa7a2ec48d6b770d04d693a1719, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/f09965bdbeb14e3386bfa9389f8d74d6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/ac6dc14859b84fd6befd9f4719a42a0d] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=52.9 K 2023-04-28 04:57:54,322 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/9310d8aa3cc349af96d0ef0788176e4a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/582603a6344a400eba005a89b40ee235, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/4e495c690f01415eb219f27c21b4b53d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5300d071889649109ce6f5a5b40143a7] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=48.1 K 2023-04-28 04:57:54,323 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting fc7d395280fe4fb584d5f3b7cc69f7e2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682657871360 2023-04-28 04:57:54,323 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9310d8aa3cc349af96d0ef0788176e4a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682657871360 2023-04-28 04:57:54,324 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 36aa3aa7a2ec48d6b770d04d693a1719, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682657872436 2023-04-28 04:57:54,324 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 582603a6344a400eba005a89b40ee235, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682657872445 2023-04-28 04:57:54,324 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f09965bdbeb14e3386bfa9389f8d74d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1682657872483 2023-04-28 04:57:54,324 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4e495c690f01415eb219f27c21b4b53d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1682657872483 2023-04-28 04:57:54,324 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ac6dc14859b84fd6befd9f4719a42a0d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682657873127 2023-04-28 04:57:54,325 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5300d071889649109ce6f5a5b40143a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682657873127 2023-04-28 04:57:54,336 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#393 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:54,336 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#392 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:54,354 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,354 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:54,355 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:54,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:54,357 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:54,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:54,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:54,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:54,357 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:54,365 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/c5f3cf13606549c581f12c6f578ddd10 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c5f3cf13606549c581f12c6f578ddd10 2023-04-28 04:57:54,369 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into c5f3cf13606549c581f12c6f578ddd10(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:54,369 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:54,369 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=12, startTime=1682657874321; duration=0sec 2023-04-28 04:57:54,369 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:54,369 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:54,370 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:54,370 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:54,370 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,370 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/60ac50343b7a4378b6efdca1dea88879, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/6ec1728020b546698de93181af256864, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/10bf7d43715643a2841d23ea8e305c48, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b9300b62d9934f7f9af83a0e8e0c4fcc] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=48.1 K 2023-04-28 04:57:54,370 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 60ac50343b7a4378b6efdca1dea88879, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682657871360 2023-04-28 04:57:54,370 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6ec1728020b546698de93181af256864, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682657872445 2023-04-28 04:57:54,371 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 10bf7d43715643a2841d23ea8e305c48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1682657872483 2023-04-28 04:57:54,371 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting b9300b62d9934f7f9af83a0e8e0c4fcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682657873127 2023-04-28 04:57:54,377 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:54,378 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,378 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:54,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:54,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,379 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,380 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:54,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:54,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:54,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:54,389 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,389 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:54,391 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:54,389 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:54,391 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:54,391 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,391 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:54,391 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:54,391 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:54,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 04:57:54,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:54,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:54,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:54,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:54,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:54,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:54,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/5874a41b72f94f138d4a4a791cc56c08 2023-04-28 04:57:54,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657934412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657934413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657934414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657934417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657934419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/7cf9f26a84f249b0b5fa4419c18dd348 2023-04-28 04:57:54,491 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:54,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:54,494 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:54,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:54,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:54,496 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:54,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:54,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:54,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:54,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,496 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,505 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:54,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:54,505 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,505 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,505 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:54,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:54,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:54,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:54,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:54,506 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:54,509 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,509 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:54,509 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:54,509 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:54,509 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:54,516 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,516 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,516 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,516 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:54,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337420aa[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:54,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:54,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,526 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,526 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,526 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:54,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:54,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:54,526 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:54,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657934523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657934523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657934525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657934526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657934527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,536 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,536 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:54,536 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:54,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:54,538 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:54,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:54,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:54,538 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:54,538 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:54,547 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,547 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,547 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,548 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,557 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:54,557 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:54,557 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:54,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:54,557 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,557 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:54,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,558 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,558 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:54,558 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:54,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:54,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:54,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:54,658 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:54,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:54,661 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:54,662 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:54,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:54,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:54,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:54,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,673 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:54,673 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:54,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,674 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:54,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:54,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:54,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:54,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:54,675 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:54,675 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:54,684 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,684 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:54,685 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:54,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4aa480[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:54,685 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,694 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:54,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:54,694 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:54,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,695 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:54,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:54,705 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:54,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:54,707 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:54,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:54,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:54,707 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:54,707 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:54,715 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,715 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,716 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:54,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:54,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:54,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:54,726 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:54,726 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:54,727 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:54,727 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:54,726 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,727 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:54,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657934729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657934730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657934730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657934733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657934733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,759 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/27adc95384884ffeba2f20f9efd0ec2f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/27adc95384884ffeba2f20f9efd0ec2f 2023-04-28 04:57:54,763 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into 27adc95384884ffeba2f20f9efd0ec2f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:54,764 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:54,764 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=12, startTime=1682657874321; duration=0sec 2023-04-28 04:57:54,764 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:54,798 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/1b38c1fe63c24bcb8e3a5a416c8b26c9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1b38c1fe63c24bcb8e3a5a416c8b26c9 2023-04-28 04:57:54,801 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into 1b38c1fe63c24bcb8e3a5a416c8b26c9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:54,801 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:54,801 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=12, startTime=1682657874321; duration=0sec 2023-04-28 04:57:54,801 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:54,827 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:54,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:54,828 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:54,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:54,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:54,829 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:54,829 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:54,829 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:54,829 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:54,830 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,830 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/eabc896b68bc4f5282c6fecb8018b59d 2023-04-28 04:57:54,860 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,860 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:54,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:54,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:54,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/5874a41b72f94f138d4a4a791cc56c08 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5874a41b72f94f138d4a4a791cc56c08 2023-04-28 04:57:54,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:54,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:54,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:54,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:54,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:54,862 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:54,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:54,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:54,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5874a41b72f94f138d4a4a791cc56c08, entries=200, sequenceid=492, filesize=14.3 K 2023-04-28 04:57:54,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/7cf9f26a84f249b0b5fa4419c18dd348 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7cf9f26a84f249b0b5fa4419c18dd348 2023-04-28 04:57:54,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7cf9f26a84f249b0b5fa4419c18dd348, entries=150, sequenceid=492, filesize=11.9 K 2023-04-28 04:57:54,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/eabc896b68bc4f5282c6fecb8018b59d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/eabc896b68bc4f5282c6fecb8018b59d 2023-04-28 04:57:54,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/eabc896b68bc4f5282c6fecb8018b59d, entries=150, sequenceid=492, filesize=11.9 K 2023-04-28 04:57:54,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 51e65c0b85e192dace926daeb81e88b4 in 484ms, sequenceid=492, compaction requested=false 2023-04-28 04:57:54,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:54,934 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:54,934 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:54,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:54,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:54,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f8a850[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:54,938 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:54,938 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,987 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:54,987 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:54,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:54,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:54,987 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:54,987 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:55,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 04:57:55,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:55,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:55,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:55,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:55,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:55,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:55,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/192834b5792a430a99584d5267a99020 2023-04-28 04:57:55,051 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:55,051 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:55,051 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:55,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657935048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:55,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:55,054 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:55,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:55,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:55,054 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:55,055 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:55,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657935049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657935050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657935051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657935052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,106 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:55,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:55,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:55,126 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,126 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:55,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:55,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657935153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657935158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657935158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657935158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657935158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,227 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:55,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:55,228 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:55,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:55,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:55,230 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:55,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:55,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:55,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:55,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,242 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:55,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:55,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:55,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:55,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:55,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:55,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:55,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:55,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:55,252 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:55,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76784415[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:55,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:55,254 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,263 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:55,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:55,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:55,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:55,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,273 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,273 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:55,273 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:55,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:55,275 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:55,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:55,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:55,275 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:55,275 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:55,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,284 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:55,284 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,285 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,295 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:55,295 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:55,295 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:55,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:55,295 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:55,295 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:55,296 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:55,297 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:55,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:55,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:55,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657935359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657935361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657935362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657935362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657935364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,395 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:55,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:55,398 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:55,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:55,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:55,399 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:55,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:55,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:55,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:55,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,410 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:55,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:55,410 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,411 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,411 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:55,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:55,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:55,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:55,423 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:55,423 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:55,423 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:55,427 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:55,427 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:55,428 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,429 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:55,429 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:55,429 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,429 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:55,429 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:55,429 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,429 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:55,429 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:55,442 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,442 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:55,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@649b4b00[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:55,444 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:55,444 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/742f701bd82742969f0a0f8636212602 2023-04-28 04:57:55,453 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:55,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:55,454 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:55,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,456 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:55,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:55,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:55,463 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:55,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:55,466 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:55,466 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:55,466 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:55,466 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:55,466 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:55,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:55,473 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:55,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,480 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:55,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:55,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:55,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:55,495 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:55,495 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:55,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:55,495 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:55,496 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:55,496 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:55,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:55,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/b1c41be9eec244e693092b80a7174c6f 2023-04-28 04:57:55,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/192834b5792a430a99584d5267a99020 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/192834b5792a430a99584d5267a99020 2023-04-28 04:57:55,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/192834b5792a430a99584d5267a99020, entries=250, sequenceid=511, filesize=16.7 K 2023-04-28 04:57:55,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/742f701bd82742969f0a0f8636212602 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/742f701bd82742969f0a0f8636212602 2023-04-28 04:57:55,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/742f701bd82742969f0a0f8636212602, entries=150, sequenceid=511, filesize=11.9 K 2023-04-28 04:57:55,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/b1c41be9eec244e693092b80a7174c6f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b1c41be9eec244e693092b80a7174c6f 2023-04-28 04:57:55,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b1c41be9eec244e693092b80a7174c6f, entries=150, sequenceid=511, filesize=11.9 K 2023-04-28 04:57:55,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 51e65c0b85e192dace926daeb81e88b4 in 497ms, sequenceid=511, compaction requested=true 2023-04-28 04:57:55,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:55,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:55,534 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:55,534 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:55,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:55,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:55,537 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44526 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:55,537 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:55,538 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,538 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c5f3cf13606549c581f12c6f578ddd10, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5874a41b72f94f138d4a4a791cc56c08, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/192834b5792a430a99584d5267a99020] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=43.5 K 2023-04-28 04:57:55,538 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:55,538 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:55,538 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,538 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/27adc95384884ffeba2f20f9efd0ec2f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7cf9f26a84f249b0b5fa4419c18dd348, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/742f701bd82742969f0a0f8636212602] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=36.3 K 2023-04-28 04:57:55,539 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 27adc95384884ffeba2f20f9efd0ec2f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682657873127 2023-04-28 04:57:55,540 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7cf9f26a84f249b0b5fa4419c18dd348, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1682657874282 2023-04-28 04:57:55,541 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c5f3cf13606549c581f12c6f578ddd10, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682657873127 2023-04-28 04:57:55,541 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 742f701bd82742969f0a0f8636212602, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682657874416 2023-04-28 04:57:55,542 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5874a41b72f94f138d4a4a791cc56c08, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1682657874282 2023-04-28 04:57:55,542 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 192834b5792a430a99584d5267a99020, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682657874412 2023-04-28 04:57:55,553 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:55,561 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:55,597 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:55,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:55,600 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:55,616 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:55,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:55,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:55,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:55,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,617 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,626 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/33e422ecd4bd40729102447afa7a69f2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/33e422ecd4bd40729102447afa7a69f2 2023-04-28 04:57:55,629 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:55,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:55,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:55,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:55,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:55,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:55,631 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:55,631 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:55,631 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:55,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:55,632 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:55,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:55,632 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 04:57:55,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:55,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:55,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:55,632 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:55,633 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:55,633 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:55,634 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into 33e422ecd4bd40729102447afa7a69f2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:55,634 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:55,634 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=13, startTime=1682657875534; duration=0sec 2023-04-28 04:57:55,634 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:55,634 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:57:55,635 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:57:55,635 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:55,635 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:55,636 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1b38c1fe63c24bcb8e3a5a416c8b26c9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/eabc896b68bc4f5282c6fecb8018b59d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b1c41be9eec244e693092b80a7174c6f] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=36.3 K 2023-04-28 04:57:55,636 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1b38c1fe63c24bcb8e3a5a416c8b26c9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682657873127 2023-04-28 04:57:55,644 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting eabc896b68bc4f5282c6fecb8018b59d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1682657874282 2023-04-28 04:57:55,645 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b1c41be9eec244e693092b80a7174c6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682657874416 2023-04-28 04:57:55,663 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#404 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:55,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:55,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:55,707 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/9a3c3bdb855144249b99841090923fd7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9a3c3bdb855144249b99841090923fd7 2023-04-28 04:57:55,718 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into 9a3c3bdb855144249b99841090923fd7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:55,718 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:55,718 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=13, startTime=1682657875534; duration=0sec 2023-04-28 04:57:55,718 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:55,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657935719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657935719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657935731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657935731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657935732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657935833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657935833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657935842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657935842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:55,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:55,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657935845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,016 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/d3ad6bd1391b45a8bb5ee47ab57f574c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d3ad6bd1391b45a8bb5ee47ab57f574c 2023-04-28 04:57:56,021 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into d3ad6bd1391b45a8bb5ee47ab57f574c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:56,021 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:56,021 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=13, startTime=1682657875534; duration=0sec 2023-04-28 04:57:56,021 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:56,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657936039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657936041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657936048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657936048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657936051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,082 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/e29857c6320c415989b26a41797221c0 2023-04-28 04:57:56,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657936348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657936349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657936355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657936356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657936356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,503 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/f9e4daedeea74edea2e05c057b3bf6d3 2023-04-28 04:57:56,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657936854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657936855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657936862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657936862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:56,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657936863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,918 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/32549c3e2b20410dad8ab2ced916e635 2023-04-28 04:57:56,921 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/e29857c6320c415989b26a41797221c0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/e29857c6320c415989b26a41797221c0 2023-04-28 04:57:56,923 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/e29857c6320c415989b26a41797221c0, entries=150, sequenceid=526, filesize=11.9 K 2023-04-28 04:57:56,924 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/f9e4daedeea74edea2e05c057b3bf6d3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/f9e4daedeea74edea2e05c057b3bf6d3 2023-04-28 04:57:56,927 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/f9e4daedeea74edea2e05c057b3bf6d3, entries=150, sequenceid=526, filesize=11.9 K 2023-04-28 04:57:56,927 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/32549c3e2b20410dad8ab2ced916e635 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/32549c3e2b20410dad8ab2ced916e635 2023-04-28 04:57:56,930 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/32549c3e2b20410dad8ab2ced916e635, entries=150, sequenceid=526, filesize=11.9 K 2023-04-28 04:57:56,930 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 51e65c0b85e192dace926daeb81e88b4 in 1298ms, sequenceid=526, compaction requested=false 2023-04-28 04:57:56,930 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:56,930 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:56,930 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:56,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:56,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:56,931 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:56,984 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:56,984 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:56,985 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:56,985 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:56,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:56,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ced9db7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:56,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:56,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:57,008 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:57,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:57,008 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:57,008 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:57,008 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:57,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,008 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:57,132 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,132 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:57,133 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:57,133 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:57,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:57,135 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:57,135 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:57,135 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:57,135 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:57,135 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:57,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:57,175 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:57,178 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:57,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:57,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:57,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:57,200 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,200 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:57,199 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:57,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:57,200 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:57,200 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:57,300 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:57,302 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:57,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:57,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:57,303 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:57,326 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:57,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:57,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:57,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:57,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:57,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:57,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:57,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:57,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:57,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:57,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:57,327 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:57,327 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:57,327 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:57,327 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 04:57:57,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:57,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:57,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:57,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:57,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:57,328 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:57,735 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/9cef182932f54dc69e62899676595213 2023-04-28 04:57:57,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:57,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:57,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657937878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657937882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657937883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657937884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657937885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657937985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657937989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657937989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657937990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:57,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:57,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657937991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,154 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/7d91fc0c278341b1876434dc68e0d8cc 2023-04-28 04:57:58,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657938191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657938197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657938198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657938198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657938198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657938497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657938502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657938502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657938503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:58,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657938503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,562 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/1c49a9cc1b4f437ba84cdcb78df6d96e 2023-04-28 04:57:58,565 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/9cef182932f54dc69e62899676595213 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/9cef182932f54dc69e62899676595213 2023-04-28 04:57:58,568 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/9cef182932f54dc69e62899676595213, entries=150, sequenceid=546, filesize=11.9 K 2023-04-28 04:57:58,569 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/7d91fc0c278341b1876434dc68e0d8cc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7d91fc0c278341b1876434dc68e0d8cc 2023-04-28 04:57:58,573 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7d91fc0c278341b1876434dc68e0d8cc, entries=150, sequenceid=546, filesize=11.9 K 2023-04-28 04:57:58,574 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/1c49a9cc1b4f437ba84cdcb78df6d96e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1c49a9cc1b4f437ba84cdcb78df6d96e 2023-04-28 04:57:58,576 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1c49a9cc1b4f437ba84cdcb78df6d96e, entries=150, sequenceid=546, filesize=11.9 K 2023-04-28 04:57:58,577 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 51e65c0b85e192dace926daeb81e88b4 in 1250ms, sequenceid=546, compaction requested=true 2023-04-28 04:57:58,577 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:58,577 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:58,577 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:58,577 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:58,577 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:58,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:58,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:58,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:58,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38931052[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:58,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,680 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,680 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:58,680 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:58,680 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:58,680 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:58,747 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:58,747 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:58,747 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,747 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:58,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:58,749 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:58,749 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:58,749 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:58,749 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:58,750 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:58,891 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,891 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:58,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,891 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:58,891 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:58,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,941 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,941 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:58,941 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:58,941 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:58,941 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:58,941 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:58,941 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:58,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:58,941 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:58,942 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:58,942 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:58,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:58,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:58,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:58,942 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:58,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:59,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:59,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 04:57:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:59,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:59,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/7ecaa0cc013240d4adf23ba2de13c56a 2023-04-28 04:57:59,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/14725b3db2284adc8cc56f3f9e4adb63 2023-04-28 04:57:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657939026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657939026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657939027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657939033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657939035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,042 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:59,044 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:59,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:59,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:59,045 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:59,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:59,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:59,045 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:59,046 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,046 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,071 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:59,071 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,071 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,071 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:59,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:59,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:59,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:59,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:59,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:59,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:59,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:59,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:59,072 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,072 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:59,072 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:59,072 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:59,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:59,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:59,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:59,083 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,083 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:59,084 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,084 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:59,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:59,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:59,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c7fa3b9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:59,085 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,094 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:59,094 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:59,094 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:59,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:59,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:59,105 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,105 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:57:59,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:59,107 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:59,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:59,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:59,107 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:59,107 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:59,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:59,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:59,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,126 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:59,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:59,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:59,126 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:59,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:57:59,126 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:59,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:59,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:59,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657939135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657939135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657939135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657939135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657939142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,226 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:59,228 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:59,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:59,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:59,229 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:59,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:59,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:59,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:59,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,241 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:59,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:59,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:59,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:59,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:59,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:59,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:59,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:57:59,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:57:59,257 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:59,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:57:59,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32823a7d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:57:59,259 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:57:59,259 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,260 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:57:59,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:57:59,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:57:59,271 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:57:59,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:57:59,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:59,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:57:59,281 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,281 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:57:59,282 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:57:59,282 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:57:59,282 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:57:59,282 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:57:59,282 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:57:59,282 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:57:59,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:57:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:57:59,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:59,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:57:59,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:59,295 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:57:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:57:59,296 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:57:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657939339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657939340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657939340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657939341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,355 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:57:59,355 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,355 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:57:59,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:57:59,355 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:57:59,355 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:57:59,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:57:59,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657939347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,356 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:57:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:57:59,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/81876dcc40214cb693f385f3c8bbe572 2023-04-28 04:57:59,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/7ecaa0cc013240d4adf23ba2de13c56a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7ecaa0cc013240d4adf23ba2de13c56a 2023-04-28 04:57:59,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7ecaa0cc013240d4adf23ba2de13c56a, entries=200, sequenceid=563, filesize=14.3 K 2023-04-28 04:57:59,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/14725b3db2284adc8cc56f3f9e4adb63 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/14725b3db2284adc8cc56f3f9e4adb63 2023-04-28 04:57:59,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/14725b3db2284adc8cc56f3f9e4adb63, entries=150, sequenceid=563, filesize=11.9 K 2023-04-28 04:57:59,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/81876dcc40214cb693f385f3c8bbe572 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/81876dcc40214cb693f385f3c8bbe572 2023-04-28 04:57:59,456 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:57:59,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:57:59,458 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:57:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:57:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:57:59,468 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:57:59,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:59,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:57:59,468 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:57:59,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/81876dcc40214cb693f385f3c8bbe572, entries=150, sequenceid=563, filesize=11.9 K 2023-04-28 04:57:59,469 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 51e65c0b85e192dace926daeb81e88b4 in 458ms, sequenceid=563, compaction requested=true 2023-04-28 04:57:59,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:59,469 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:59,470 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:59,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:59,470 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:59,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:59,471 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:59,471 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/B is initiating minor compaction (all files) 2023-04-28 04:57:59,471 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/B in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,471 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/33e422ecd4bd40729102447afa7a69f2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/f9e4daedeea74edea2e05c057b3bf6d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7d91fc0c278341b1876434dc68e0d8cc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/14725b3db2284adc8cc56f3f9e4adb63] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=48.3 K 2023-04-28 04:57:59,472 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:59,472 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 33e422ecd4bd40729102447afa7a69f2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682657874416 2023-04-28 04:57:59,472 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/A is initiating minor compaction (all files) 2023-04-28 04:57:59,472 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/A in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,472 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d3ad6bd1391b45a8bb5ee47ab57f574c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/e29857c6320c415989b26a41797221c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/9cef182932f54dc69e62899676595213, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7ecaa0cc013240d4adf23ba2de13c56a] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=50.7 K 2023-04-28 04:57:59,472 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting f9e4daedeea74edea2e05c057b3bf6d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682657875047 2023-04-28 04:57:59,472 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7d91fc0c278341b1876434dc68e0d8cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1682657875728 2023-04-28 04:57:59,472 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting d3ad6bd1391b45a8bb5ee47ab57f574c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682657874416 2023-04-28 04:57:59,473 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 14725b3db2284adc8cc56f3f9e4adb63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682657877883 2023-04-28 04:57:59,473 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting e29857c6320c415989b26a41797221c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682657875047 2023-04-28 04:57:59,473 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9cef182932f54dc69e62899676595213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1682657875728 2023-04-28 04:57:59,474 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7ecaa0cc013240d4adf23ba2de13c56a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682657877882 2023-04-28 04:57:59,481 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#B#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:59,485 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#A#compaction#414 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:59,494 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:57:59,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:57:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:57:59,494 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,494 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:57:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:57:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:57:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:57:59,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:57:59,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:57:59,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:57:59,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:57:59,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:57:59,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:57:59,504 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:57:59,504 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. started... 2023-04-28 04:57:59,504 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:57:59,504 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:57:59,509 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/424158cdf26443e3a99fe253c9d54bd7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/424158cdf26443e3a99fe253c9d54bd7 2023-04-28 04:57:59,513 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/A of 51e65c0b85e192dace926daeb81e88b4 into 424158cdf26443e3a99fe253c9d54bd7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:59,513 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:59,513 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/A, priority=12, startTime=1682657879469; duration=0sec 2023-04-28 04:57:59,513 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:57:59,513 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:57:59,514 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:57:59,514 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): 51e65c0b85e192dace926daeb81e88b4/C is initiating minor compaction (all files) 2023-04-28 04:57:59,514 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e65c0b85e192dace926daeb81e88b4/C in TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:57:59,514 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9a3c3bdb855144249b99841090923fd7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/32549c3e2b20410dad8ab2ced916e635, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1c49a9cc1b4f437ba84cdcb78df6d96e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/81876dcc40214cb693f385f3c8bbe572] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp, totalSize=48.3 K 2023-04-28 04:57:59,515 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 9a3c3bdb855144249b99841090923fd7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682657874416 2023-04-28 04:57:59,515 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 32549c3e2b20410dad8ab2ced916e635, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682657875047 2023-04-28 04:57:59,515 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 1c49a9cc1b4f437ba84cdcb78df6d96e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1682657875728 2023-04-28 04:57:59,515 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 81876dcc40214cb693f385f3c8bbe572, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682657877883 2023-04-28 04:57:59,527 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e65c0b85e192dace926daeb81e88b4#C#compaction#416 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:57:59,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:57:59,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. as already flushing 2023-04-28 04:57:59,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657939665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657939669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657939670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657939675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657939676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657939777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657939777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657939777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657939782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657939783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,896 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/b7afac2baaf7497d87f8b22b04c88f1b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/b7afac2baaf7497d87f8b22b04c88f1b 2023-04-28 04:57:59,899 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/B of 51e65c0b85e192dace926daeb81e88b4 into b7afac2baaf7497d87f8b22b04c88f1b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:59,900 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:59,900 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/B, priority=12, startTime=1682657879470; duration=0sec 2023-04-28 04:57:59,900 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:59,932 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/d224d34cd26349478cb39b75c38dac5d 2023-04-28 04:57:59,937 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/f4429fcf49a54e16a610fc5c7e0b6e71 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f4429fcf49a54e16a610fc5c7e0b6e71 2023-04-28 04:57:59,947 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e65c0b85e192dace926daeb81e88b4/C of 51e65c0b85e192dace926daeb81e88b4 into f4429fcf49a54e16a610fc5c7e0b6e71(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:57:59,947 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:57:59,947 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4., storeName=51e65c0b85e192dace926daeb81e88b4/C, priority=12, startTime=1682657879470; duration=0sec 2023-04-28 04:57:59,947 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:57:59,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657939983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657939983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657939984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657939987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:57:59,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:57:59,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657939989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:00,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57246 deadline: 1682657940287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:00,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57220 deadline: 1682657940288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:00,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57254 deadline: 1682657940289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:00,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57162 deadline: 1682657940295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:00,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:57196 deadline: 1682657940296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,354 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/42c264a5eecd469f8ecd0cd204bb211e 2023-04-28 04:58:00,464 DEBUG [Thread-2628] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4486b3c7 to 127.0.0.1:52932 2023-04-28 04:58:00,464 DEBUG [Thread-2628] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,464 DEBUG [Thread-2624] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79db57d4 to 127.0.0.1:52932 2023-04-28 04:58:00,464 DEBUG [Thread-2624] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,466 DEBUG [Thread-2622] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fc0e3f to 127.0.0.1:52932 2023-04-28 04:58:00,466 DEBUG [Thread-2622] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,469 DEBUG [Thread-2626] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e56ea94 to 127.0.0.1:52932 2023-04-28 04:58:00,469 DEBUG [Thread-2626] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,469 DEBUG [Thread-2630] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x149f1de0 to 127.0.0.1:52932 2023-04-28 04:58:00,469 DEBUG [Thread-2630] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,763 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/c486cff708954a04a8d806a862dbf85b 2023-04-28 04:58:00,766 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/d224d34cd26349478cb39b75c38dac5d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d224d34cd26349478cb39b75c38dac5d 2023-04-28 04:58:00,769 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d224d34cd26349478cb39b75c38dac5d, entries=150, sequenceid=579, filesize=11.9 K 2023-04-28 04:58:00,770 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/42c264a5eecd469f8ecd0cd204bb211e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/42c264a5eecd469f8ecd0cd204bb211e 2023-04-28 04:58:00,773 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/42c264a5eecd469f8ecd0cd204bb211e, entries=150, sequenceid=579, filesize=11.9 K 2023-04-28 04:58:00,774 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/c486cff708954a04a8d806a862dbf85b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c486cff708954a04a8d806a862dbf85b 2023-04-28 04:58:00,778 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c486cff708954a04a8d806a862dbf85b, entries=150, sequenceid=579, filesize=11.9 K 2023-04-28 04:58:00,779 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 51e65c0b85e192dace926daeb81e88b4 in 1275ms, sequenceid=579, compaction requested=false 2023-04-28 04:58:00,779 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:58:00,779 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:58:00,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:00,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:00,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:00,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:00,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:00,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,789 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:00,790 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@221209bf[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:00,791 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:58:00,793 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 04:58:00,793 DEBUG [Thread-2613] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41ccedcb to 127.0.0.1:52932 2023-04-28 04:58:00,793 DEBUG [Thread-2613] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:58:00,793 DEBUG [Thread-2611] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cb7a788 to 127.0.0.1:52932 2023-04-28 04:58:00,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:00,794 DEBUG [Thread-2611] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:58:00,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:00,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:58:00,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:00,795 DEBUG [Thread-2617] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5d7fcf to 127.0.0.1:52932 2023-04-28 04:58:00,795 DEBUG [Thread-2617] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:00,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:00,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:00,804 DEBUG [Thread-2615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e6f624 to 127.0.0.1:52932 2023-04-28 04:58:00,804 DEBUG [Thread-2615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,805 DEBUG [Thread-2619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ea9d51a to 127.0.0.1:52932 2023-04-28 04:58:00,805 DEBUG [Thread-2619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,810 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,810 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:00,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:00,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:00,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,831 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:00,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:00,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:00,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:00,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:00,834 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:00,834 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:00,841 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,841 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:00,841 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,842 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:00,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:00,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,852 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:00,852 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:00,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:00,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:00,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:00,852 DEBUG [Thread-2621] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:00,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:00,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:00,852 DEBUG [Thread-2621] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:00,852 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:00,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:00,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:00,853 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:00,953 DEBUG [Thread-2621] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:00,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 95 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 95 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 04:58:00,954 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1957 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5871 rows 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1960 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5880 rows 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1940 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5816 rows 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1954 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5856 rows 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(402): scanned 1962 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(403): verified 5886 rows 2023-04-28 04:58:00,955 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:58:00,955 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02e8c6f6 to 127.0.0.1:52932 2023-04-28 04:58:00,955 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:00,956 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 04:58:00,957 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$11(2418): Client=jenkins//136.243.104.168 disable TestAcidGuarantees 2023-04-28 04:58:00,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:00,960 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657880960"}]},"ts":"1682657880960"} 2023-04-28 04:58:00,961 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 04:58:00,978 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 04:58:00,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, UNASSIGN}] 2023-04-28 04:58:00,980 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, UNASSIGN 2023-04-28 04:58:00,981 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=51e65c0b85e192dace926daeb81e88b4, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:00,981 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657880981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657880981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657880981"}]},"ts":"1682657880981"} 2023-04-28 04:58:00,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:58:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:01,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:58:01,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 51e65c0b85e192dace926daeb81e88b4, disabling compactions & flushes 2023-04-28 04:58:01,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:58:01,199 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/276c376c0d2b46b39174d044097d1252 2023-04-28 04:58:01,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:01,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:01,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/96e46db4438a472b8d1df52848f4d5f5 2023-04-28 04:58:02,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/9e04f5da037b4d1cb0078cc35f921c3a 2023-04-28 04:58:02,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/276c376c0d2b46b39174d044097d1252 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/276c376c0d2b46b39174d044097d1252 2023-04-28 04:58:02,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/276c376c0d2b46b39174d044097d1252, entries=150, sequenceid=601, filesize=11.9 K 2023-04-28 04:58:02,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/96e46db4438a472b8d1df52848f4d5f5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/96e46db4438a472b8d1df52848f4d5f5 2023-04-28 04:58:02,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/96e46db4438a472b8d1df52848f4d5f5, entries=150, sequenceid=601, filesize=11.9 K 2023-04-28 04:58:02,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/9e04f5da037b4d1cb0078cc35f921c3a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9e04f5da037b4d1cb0078cc35f921c3a 2023-04-28 04:58:02,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9e04f5da037b4d1cb0078cc35f921c3a, entries=150, sequenceid=601, filesize=11.9 K 2023-04-28 04:58:02,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=20.13 KB/20610 for 51e65c0b85e192dace926daeb81e88b4 in 1244ms, sequenceid=601, compaction requested=true 2023-04-28 04:58:02,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:58:02,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:58:02,038 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. because compaction request was cancelled 2023-04-28 04:58:02,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:58:02,038 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. because compaction request was cancelled 2023-04-28 04:58:02,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. after waiting 0 ms 2023-04-28 04:58:02,038 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. because compaction request was cancelled 2023-04-28 04:58:02,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:58:02,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing 51e65c0b85e192dace926daeb81e88b4 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=A 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=B 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e65c0b85e192dace926daeb81e88b4, store=C 2023-04-28 04:58:02,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:02,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/c1e64f28413448b6b106c15cd97bfa57 2023-04-28 04:58:02,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/302c14cde5e3449d8bd85c9f6c29efb6 2023-04-28 04:58:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:03,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/f8cf2e123e61413ca151f7f4ea525b5a 2023-04-28 04:58:03,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/A/c1e64f28413448b6b106c15cd97bfa57 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c1e64f28413448b6b106c15cd97bfa57 2023-04-28 04:58:03,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c1e64f28413448b6b106c15cd97bfa57, entries=100, sequenceid=607, filesize=9.5 K 2023-04-28 04:58:03,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/B/302c14cde5e3449d8bd85c9f6c29efb6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/302c14cde5e3449d8bd85c9f6c29efb6 2023-04-28 04:58:03,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/302c14cde5e3449d8bd85c9f6c29efb6, entries=100, sequenceid=607, filesize=9.5 K 2023-04-28 04:58:03,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/.tmp/C/f8cf2e123e61413ca151f7f4ea525b5a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f8cf2e123e61413ca151f7f4ea525b5a 2023-04-28 04:58:03,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f8cf2e123e61413ca151f7f4ea525b5a, entries=100, sequenceid=607, filesize=9.5 K 2023-04-28 04:58:03,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 51e65c0b85e192dace926daeb81e88b4 in 1259ms, sequenceid=607, compaction requested=true 2023-04-28 04:58:03,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/337c0572728b41c4b961fdec3ecc71f2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5e260ab6f1f74242b7ccf8df3905f6a8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d66ee2e1a1b140a39c9dad4643fa6dcf, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0480db5177524d37bea328734f57f8fb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b51dde8a5be840148a380a050f24082d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/19cac5128a52447780682ef53931feeb, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0def8771bc294dceacbb683f37b40c63, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b347cdc193a84d6d8a8edbdca0bd5ba8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b414a70730fa4a9aa712ec1db7a927f7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/6e75bc4cd0414f448b30714b848284b9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/4fb08ba38faf40f6b262e47ac790305e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/2bf6fac20b1e4f628edc1a224ca6b4bc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/166d0621789a4634ba56c3bd0a118376, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7e77f33783f443359396855b616b2144, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/01cb7211ad6046f2b079088f9b1e5a17, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/fc7d395280fe4fb584d5f3b7cc69f7e2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/36aa3aa7a2ec48d6b770d04d693a1719, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/f09965bdbeb14e3386bfa9389f8d74d6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/ac6dc14859b84fd6befd9f4719a42a0d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c5f3cf13606549c581f12c6f578ddd10, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5874a41b72f94f138d4a4a791cc56c08, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/192834b5792a430a99584d5267a99020, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d3ad6bd1391b45a8bb5ee47ab57f574c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/e29857c6320c415989b26a41797221c0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/9cef182932f54dc69e62899676595213, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7ecaa0cc013240d4adf23ba2de13c56a] to archive 2023-04-28 04:58:03,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:58:03,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/337c0572728b41c4b961fdec3ecc71f2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/337c0572728b41c4b961fdec3ecc71f2 2023-04-28 04:58:03,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5e260ab6f1f74242b7ccf8df3905f6a8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5e260ab6f1f74242b7ccf8df3905f6a8 2023-04-28 04:58:03,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d66ee2e1a1b140a39c9dad4643fa6dcf to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d66ee2e1a1b140a39c9dad4643fa6dcf 2023-04-28 04:58:03,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0480db5177524d37bea328734f57f8fb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0480db5177524d37bea328734f57f8fb 2023-04-28 04:58:03,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b51dde8a5be840148a380a050f24082d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b51dde8a5be840148a380a050f24082d 2023-04-28 04:58:03,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/19cac5128a52447780682ef53931feeb to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/19cac5128a52447780682ef53931feeb 2023-04-28 04:58:03,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0def8771bc294dceacbb683f37b40c63 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/0def8771bc294dceacbb683f37b40c63 2023-04-28 04:58:03,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b347cdc193a84d6d8a8edbdca0bd5ba8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b347cdc193a84d6d8a8edbdca0bd5ba8 2023-04-28 04:58:03,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b414a70730fa4a9aa712ec1db7a927f7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/b414a70730fa4a9aa712ec1db7a927f7 2023-04-28 04:58:03,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/6e75bc4cd0414f448b30714b848284b9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/6e75bc4cd0414f448b30714b848284b9 2023-04-28 04:58:03,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/4fb08ba38faf40f6b262e47ac790305e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/4fb08ba38faf40f6b262e47ac790305e 2023-04-28 04:58:03,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/2bf6fac20b1e4f628edc1a224ca6b4bc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/2bf6fac20b1e4f628edc1a224ca6b4bc 2023-04-28 04:58:03,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/166d0621789a4634ba56c3bd0a118376 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/166d0621789a4634ba56c3bd0a118376 2023-04-28 04:58:03,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7e77f33783f443359396855b616b2144 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7e77f33783f443359396855b616b2144 2023-04-28 04:58:03,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/01cb7211ad6046f2b079088f9b1e5a17 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/01cb7211ad6046f2b079088f9b1e5a17 2023-04-28 04:58:03,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/fc7d395280fe4fb584d5f3b7cc69f7e2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/fc7d395280fe4fb584d5f3b7cc69f7e2 2023-04-28 04:58:03,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/36aa3aa7a2ec48d6b770d04d693a1719 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/36aa3aa7a2ec48d6b770d04d693a1719 2023-04-28 04:58:03,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/f09965bdbeb14e3386bfa9389f8d74d6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/f09965bdbeb14e3386bfa9389f8d74d6 2023-04-28 04:58:03,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/ac6dc14859b84fd6befd9f4719a42a0d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/ac6dc14859b84fd6befd9f4719a42a0d 2023-04-28 04:58:03,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c5f3cf13606549c581f12c6f578ddd10 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c5f3cf13606549c581f12c6f578ddd10 2023-04-28 04:58:03,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5874a41b72f94f138d4a4a791cc56c08 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/5874a41b72f94f138d4a4a791cc56c08 2023-04-28 04:58:03,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/192834b5792a430a99584d5267a99020 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/192834b5792a430a99584d5267a99020 2023-04-28 04:58:03,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d3ad6bd1391b45a8bb5ee47ab57f574c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d3ad6bd1391b45a8bb5ee47ab57f574c 2023-04-28 04:58:03,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/e29857c6320c415989b26a41797221c0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/e29857c6320c415989b26a41797221c0 2023-04-28 04:58:03,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/9cef182932f54dc69e62899676595213 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/9cef182932f54dc69e62899676595213 2023-04-28 04:58:03,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7ecaa0cc013240d4adf23ba2de13c56a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/7ecaa0cc013240d4adf23ba2de13c56a 2023-04-28 04:58:03,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a4114407c6fb491cab475b9375fbe853, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/c718258e522840d1bad8ef55a74e9d51, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/20e4a1f56c3c420594c077b1f1d5fab4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/947adc8f73c94dfabfcf9b3f66e8296b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/15b05d1f5c3a492a87330adf7706bca1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5d921e8fedd24fc3b1f81deba76d9750, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/034b3a85cfa94f0083fc0f40ca914629, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/797250f6e6214564b39263cec59ffcd7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/2f9205ef3d89464698acc8340f9bfca6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a92fabb7c87e4fe8b68af46715b5fea9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/3afed8d39f5d4478afc315235a952e67, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/90f7da913bb04203bff357bee6006566, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/e475fa287d384434b670584d9f076472, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/91595bf705264f339d7ab2f77169e4a6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/9310d8aa3cc349af96d0ef0788176e4a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/41dae963b188461090afb28cf197f7b3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/582603a6344a400eba005a89b40ee235, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/4e495c690f01415eb219f27c21b4b53d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/27adc95384884ffeba2f20f9efd0ec2f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5300d071889649109ce6f5a5b40143a7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7cf9f26a84f249b0b5fa4419c18dd348, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/33e422ecd4bd40729102447afa7a69f2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/742f701bd82742969f0a0f8636212602, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/f9e4daedeea74edea2e05c057b3bf6d3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7d91fc0c278341b1876434dc68e0d8cc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/14725b3db2284adc8cc56f3f9e4adb63] to archive 2023-04-28 04:58:03,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:58:03,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a4114407c6fb491cab475b9375fbe853 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a4114407c6fb491cab475b9375fbe853 2023-04-28 04:58:03,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/c718258e522840d1bad8ef55a74e9d51 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/c718258e522840d1bad8ef55a74e9d51 2023-04-28 04:58:03,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/20e4a1f56c3c420594c077b1f1d5fab4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/20e4a1f56c3c420594c077b1f1d5fab4 2023-04-28 04:58:03,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/947adc8f73c94dfabfcf9b3f66e8296b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/947adc8f73c94dfabfcf9b3f66e8296b 2023-04-28 04:58:03,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/15b05d1f5c3a492a87330adf7706bca1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/15b05d1f5c3a492a87330adf7706bca1 2023-04-28 04:58:03,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5d921e8fedd24fc3b1f81deba76d9750 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5d921e8fedd24fc3b1f81deba76d9750 2023-04-28 04:58:03,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/034b3a85cfa94f0083fc0f40ca914629 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/034b3a85cfa94f0083fc0f40ca914629 2023-04-28 04:58:03,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/797250f6e6214564b39263cec59ffcd7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/797250f6e6214564b39263cec59ffcd7 2023-04-28 04:58:03,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/2f9205ef3d89464698acc8340f9bfca6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/2f9205ef3d89464698acc8340f9bfca6 2023-04-28 04:58:03,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a92fabb7c87e4fe8b68af46715b5fea9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/a92fabb7c87e4fe8b68af46715b5fea9 2023-04-28 04:58:03,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/3afed8d39f5d4478afc315235a952e67 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/3afed8d39f5d4478afc315235a952e67 2023-04-28 04:58:03,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/90f7da913bb04203bff357bee6006566 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/90f7da913bb04203bff357bee6006566 2023-04-28 04:58:03,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/e475fa287d384434b670584d9f076472 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/e475fa287d384434b670584d9f076472 2023-04-28 04:58:03,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/91595bf705264f339d7ab2f77169e4a6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/91595bf705264f339d7ab2f77169e4a6 2023-04-28 04:58:03,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/9310d8aa3cc349af96d0ef0788176e4a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/9310d8aa3cc349af96d0ef0788176e4a 2023-04-28 04:58:03,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/41dae963b188461090afb28cf197f7b3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/41dae963b188461090afb28cf197f7b3 2023-04-28 04:58:03,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/582603a6344a400eba005a89b40ee235 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/582603a6344a400eba005a89b40ee235 2023-04-28 04:58:03,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/4e495c690f01415eb219f27c21b4b53d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/4e495c690f01415eb219f27c21b4b53d 2023-04-28 04:58:03,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/27adc95384884ffeba2f20f9efd0ec2f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/27adc95384884ffeba2f20f9efd0ec2f 2023-04-28 04:58:03,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5300d071889649109ce6f5a5b40143a7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/5300d071889649109ce6f5a5b40143a7 2023-04-28 04:58:03,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7cf9f26a84f249b0b5fa4419c18dd348 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7cf9f26a84f249b0b5fa4419c18dd348 2023-04-28 04:58:03,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/33e422ecd4bd40729102447afa7a69f2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/33e422ecd4bd40729102447afa7a69f2 2023-04-28 04:58:03,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/742f701bd82742969f0a0f8636212602 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/742f701bd82742969f0a0f8636212602 2023-04-28 04:58:03,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/f9e4daedeea74edea2e05c057b3bf6d3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/f9e4daedeea74edea2e05c057b3bf6d3 2023-04-28 04:58:03,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7d91fc0c278341b1876434dc68e0d8cc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/7d91fc0c278341b1876434dc68e0d8cc 2023-04-28 04:58:03,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/14725b3db2284adc8cc56f3f9e4adb63 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/14725b3db2284adc8cc56f3f9e4adb63 2023-04-28 04:58:03,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/89e76aad9f8b4b858aba3595f0e1a5f6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f44f3e4073244c31bc6a517504e88d37, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/d01ee5c77ba74127be727e7ecd0f211a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/38182ce692934f2c8984db3b9cc85a5a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/612853b2f67a4eb088d36ac51e57cd72, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/bcdbf3f2ed394ff09e323f7dc9ebd025, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c818f9af430a4af3a5f2f50ecf157bf3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/22e0f42ac023423bab8507b28d9c2205, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/649bf87931fa437d9b9589787d05014e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/849d761235744ae1a97988642171fa07, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/a6f36e74b4fe452283885be65606f255, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/772cf5572ae240f7b5a814dca9f93de7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/7e7ff484a81c411987a62a45557c8929, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/cd6ceee68e0a4c4b9dd41ab7b56063f6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/60ac50343b7a4378b6efdca1dea88879, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/e0b189b6ec1c44f09d3668b3ef34b3c6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/6ec1728020b546698de93181af256864, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/10bf7d43715643a2841d23ea8e305c48, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1b38c1fe63c24bcb8e3a5a416c8b26c9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b9300b62d9934f7f9af83a0e8e0c4fcc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/eabc896b68bc4f5282c6fecb8018b59d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9a3c3bdb855144249b99841090923fd7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b1c41be9eec244e693092b80a7174c6f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/32549c3e2b20410dad8ab2ced916e635, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1c49a9cc1b4f437ba84cdcb78df6d96e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/81876dcc40214cb693f385f3c8bbe572] to archive 2023-04-28 04:58:03,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:58:03,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/89e76aad9f8b4b858aba3595f0e1a5f6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/89e76aad9f8b4b858aba3595f0e1a5f6 2023-04-28 04:58:03,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f44f3e4073244c31bc6a517504e88d37 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f44f3e4073244c31bc6a517504e88d37 2023-04-28 04:58:03,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/d01ee5c77ba74127be727e7ecd0f211a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/d01ee5c77ba74127be727e7ecd0f211a 2023-04-28 04:58:03,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/38182ce692934f2c8984db3b9cc85a5a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/38182ce692934f2c8984db3b9cc85a5a 2023-04-28 04:58:03,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/612853b2f67a4eb088d36ac51e57cd72 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/612853b2f67a4eb088d36ac51e57cd72 2023-04-28 04:58:03,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/bcdbf3f2ed394ff09e323f7dc9ebd025 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/bcdbf3f2ed394ff09e323f7dc9ebd025 2023-04-28 04:58:03,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c818f9af430a4af3a5f2f50ecf157bf3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c818f9af430a4af3a5f2f50ecf157bf3 2023-04-28 04:58:03,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/22e0f42ac023423bab8507b28d9c2205 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/22e0f42ac023423bab8507b28d9c2205 2023-04-28 04:58:03,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/649bf87931fa437d9b9589787d05014e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/649bf87931fa437d9b9589787d05014e 2023-04-28 04:58:03,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/849d761235744ae1a97988642171fa07 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/849d761235744ae1a97988642171fa07 2023-04-28 04:58:03,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/a6f36e74b4fe452283885be65606f255 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/a6f36e74b4fe452283885be65606f255 2023-04-28 04:58:03,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/772cf5572ae240f7b5a814dca9f93de7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/772cf5572ae240f7b5a814dca9f93de7 2023-04-28 04:58:03,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/7e7ff484a81c411987a62a45557c8929 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/7e7ff484a81c411987a62a45557c8929 2023-04-28 04:58:03,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/cd6ceee68e0a4c4b9dd41ab7b56063f6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/cd6ceee68e0a4c4b9dd41ab7b56063f6 2023-04-28 04:58:03,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/60ac50343b7a4378b6efdca1dea88879 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/60ac50343b7a4378b6efdca1dea88879 2023-04-28 04:58:03,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/e0b189b6ec1c44f09d3668b3ef34b3c6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/e0b189b6ec1c44f09d3668b3ef34b3c6 2023-04-28 04:58:03,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/6ec1728020b546698de93181af256864 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/6ec1728020b546698de93181af256864 2023-04-28 04:58:03,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/10bf7d43715643a2841d23ea8e305c48 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/10bf7d43715643a2841d23ea8e305c48 2023-04-28 04:58:03,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1b38c1fe63c24bcb8e3a5a416c8b26c9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1b38c1fe63c24bcb8e3a5a416c8b26c9 2023-04-28 04:58:03,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b9300b62d9934f7f9af83a0e8e0c4fcc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b9300b62d9934f7f9af83a0e8e0c4fcc 2023-04-28 04:58:03,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/eabc896b68bc4f5282c6fecb8018b59d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/eabc896b68bc4f5282c6fecb8018b59d 2023-04-28 04:58:03,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9a3c3bdb855144249b99841090923fd7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9a3c3bdb855144249b99841090923fd7 2023-04-28 04:58:03,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b1c41be9eec244e693092b80a7174c6f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/b1c41be9eec244e693092b80a7174c6f 2023-04-28 04:58:03,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/32549c3e2b20410dad8ab2ced916e635 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/32549c3e2b20410dad8ab2ced916e635 2023-04-28 04:58:03,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1c49a9cc1b4f437ba84cdcb78df6d96e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/1c49a9cc1b4f437ba84cdcb78df6d96e 2023-04-28 04:58:03,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/81876dcc40214cb693f385f3c8bbe572 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/81876dcc40214cb693f385f3c8bbe572 2023-04-28 04:58:03,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/recovered.edits/610.seqid, newMaxSeqId=610, maxSeqId=1 2023-04-28 04:58:03,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4. 2023-04-28 04:58:03,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 51e65c0b85e192dace926daeb81e88b4: 2023-04-28 04:58:03,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed 51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:58:03,406 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=51e65c0b85e192dace926daeb81e88b4, regionState=CLOSED 2023-04-28 04:58:03,406 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657883406"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657883406"}]},"ts":"1682657883406"} 2023-04-28 04:58:03,408 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-28 04:58:03,409 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 51e65c0b85e192dace926daeb81e88b4, server=jenkins-hbase12.apache.org,37663,1682657733313 in 2.4250 sec 2023-04-28 04:58:03,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-28 04:58:03,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e65c0b85e192dace926daeb81e88b4, UNASSIGN in 2.4290 sec 2023-04-28 04:58:03,411 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657883411"}]},"ts":"1682657883411"} 2023-04-28 04:58:03,412 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 04:58:03,420 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 04:58:03,422 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4640 sec 2023-04-28 04:58:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 04:58:05,066 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-28 04:58:05,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$5(2228): Client=jenkins//136.243.104.168 delete TestAcidGuarantees 2023-04-28 04:58:05,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,069 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,070 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-28 04:58:05,075 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:58:05,079 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/recovered.edits] 2023-04-28 04:58:05,084 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/276c376c0d2b46b39174d044097d1252 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/276c376c0d2b46b39174d044097d1252 2023-04-28 04:58:05,086 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/424158cdf26443e3a99fe253c9d54bd7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/424158cdf26443e3a99fe253c9d54bd7 2023-04-28 04:58:05,088 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c1e64f28413448b6b106c15cd97bfa57 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/c1e64f28413448b6b106c15cd97bfa57 2023-04-28 04:58:05,089 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d224d34cd26349478cb39b75c38dac5d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/A/d224d34cd26349478cb39b75c38dac5d 2023-04-28 04:58:05,092 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/302c14cde5e3449d8bd85c9f6c29efb6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/302c14cde5e3449d8bd85c9f6c29efb6 2023-04-28 04:58:05,093 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/42c264a5eecd469f8ecd0cd204bb211e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/42c264a5eecd469f8ecd0cd204bb211e 2023-04-28 04:58:05,094 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/96e46db4438a472b8d1df52848f4d5f5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/96e46db4438a472b8d1df52848f4d5f5 2023-04-28 04:58:05,095 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/b7afac2baaf7497d87f8b22b04c88f1b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/B/b7afac2baaf7497d87f8b22b04c88f1b 2023-04-28 04:58:05,097 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9e04f5da037b4d1cb0078cc35f921c3a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/9e04f5da037b4d1cb0078cc35f921c3a 2023-04-28 04:58:05,098 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c486cff708954a04a8d806a862dbf85b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/c486cff708954a04a8d806a862dbf85b 2023-04-28 04:58:05,098 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f4429fcf49a54e16a610fc5c7e0b6e71 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f4429fcf49a54e16a610fc5c7e0b6e71 2023-04-28 04:58:05,099 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f8cf2e123e61413ca151f7f4ea525b5a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/C/f8cf2e123e61413ca151f7f4ea525b5a 2023-04-28 04:58:05,101 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/recovered.edits/610.seqid to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4/recovered.edits/610.seqid 2023-04-28 04:58:05,102 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/51e65c0b85e192dace926daeb81e88b4 2023-04-28 04:58:05,102 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:58:05,103 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,105 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 04:58:05,106 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 04:58:05,107 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,107 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 04:58:05,108 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682657885107"}]},"ts":"9223372036854775807"} 2023-04-28 04:58:05,109 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 04:58:05,109 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 51e65c0b85e192dace926daeb81e88b4, NAME => 'TestAcidGuarantees,,1682657859100.51e65c0b85e192dace926daeb81e88b4.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 04:58:05,109 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 04:58:05,109 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682657885109"}]},"ts":"9223372036854775807"} 2023-04-28 04:58:05,111 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 04:58:05,169 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,171 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 102 msec 2023-04-28 04:58:05,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-28 04:58:05,173 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-28 04:58:05,184 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 229), OpenFileDescriptor=562 (was 527) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=505 (was 497) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=10275 (was 9965) - AvailableMemoryMB LEAK? - 2023-04-28 04:58:05,197 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=562, MaxFileDescriptor=60000, SystemLoadAverage=505, ProcessCount=173, AvailableMemoryMB=10275 2023-04-28 04:58:05,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:58:05,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$4(2112): Client=jenkins//136.243.104.168 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:58:05,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:05,202 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 04:58:05,202 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(697): Client=jenkins//136.243.104.168 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-28 04:58:05,203 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 04:58:05,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 04:58:05,204 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:05,204 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 empty. 2023-04-28 04:58:05,204 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:05,204 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:58:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 04:58:05,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 04:58:05,612 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:58:05,615 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a6bab1698a9b263a4df9a801810dbac0, NAME => 'TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp 2023-04-28 04:58:05,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 04:58:06,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:58:06,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a6bab1698a9b263a4df9a801810dbac0, disabling compactions & flushes 2023-04-28 04:58:06,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. after waiting 0 ms 2023-04-28 04:58:06,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:06,028 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 04:58:06,029 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657886029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657886029"}]},"ts":"1682657886029"} 2023-04-28 04:58:06,030 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 04:58:06,031 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 04:58:06,031 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657886031"}]},"ts":"1682657886031"} 2023-04-28 04:58:06,032 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 04:58:06,052 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, ASSIGN}] 2023-04-28 04:58:06,054 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, ASSIGN 2023-04-28 04:58:06,054 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, ASSIGN; state=OFFLINE, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=false 2023-04-28 04:58:06,206 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:06,206 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657886206"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657886206"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657886206"}]},"ts":"1682657886206"} 2023-04-28 04:58:06,208 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:58:06,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 04:58:06,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a6bab1698a9b263a4df9a801810dbac0, NAME => 'TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:58:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:58:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,367 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,369 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A 2023-04-28 04:58:06,369 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A 2023-04-28 04:58:06,369 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:58:06,370 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6bab1698a9b263a4df9a801810dbac0 columnFamilyName A 2023-04-28 04:58:06,370 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(310): Store=a6bab1698a9b263a4df9a801810dbac0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:58:06,371 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,373 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B 2023-04-28 04:58:06,373 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B 2023-04-28 04:58:06,373 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:58:06,373 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6bab1698a9b263a4df9a801810dbac0 columnFamilyName B 2023-04-28 04:58:06,374 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(310): Store=a6bab1698a9b263a4df9a801810dbac0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:58:06,374 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,376 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C 2023-04-28 04:58:06,376 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C 2023-04-28 04:58:06,377 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:58:06,377 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6bab1698a9b263a4df9a801810dbac0 columnFamilyName C 2023-04-28 04:58:06,378 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(310): Store=a6bab1698a9b263a4df9a801810dbac0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:58:06,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:58:06,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:06,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 04:58:06,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened a6bab1698a9b263a4df9a801810dbac0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69582248, jitterRate=0.03685629367828369}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:58:06,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:06,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., pid=56, masterSystemTime=1682657886360 2023-04-28 04:58:06,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:06,390 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:06,391 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657886390"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657886390"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657886390"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657886390"}]},"ts":"1682657886390"} 2023-04-28 04:58:06,394 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-28 04:58:06,394 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 184 msec 2023-04-28 04:58:06,397 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-28 04:58:06,397 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, ASSIGN in 343 msec 2023-04-28 04:58:06,398 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 04:58:06,398 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657886398"}]},"ts":"1682657886398"} 2023-04-28 04:58:06,400 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 04:58:06,411 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 04:58:06,413 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2110 sec 2023-04-28 04:58:07,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 04:58:07,311 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-28 04:58:07,312 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x5be55982 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:07,327 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@630ea023, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:07,330 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:07,333 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:07,336 DEBUG [Listener at localhost.localdomain/36983] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 04:58:07,339 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:35968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 04:58:07,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 04:58:07,341 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$12(2454): Client=jenkins//136.243.104.168 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 04:58:07,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:07,752 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 04:58:07,754 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 04:58:07,754 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 04:58:07,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 04:58:07,759 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, REOPEN/MOVE}] 2023-04-28 04:58:07,760 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, REOPEN/MOVE 2023-04-28 04:58:07,760 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:07,761 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657887760"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657887760"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657887760"}]},"ts":"1682657887760"} 2023-04-28 04:58:07,762 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:58:07,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:07,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing a6bab1698a9b263a4df9a801810dbac0, disabling compactions & flushes 2023-04-28 04:58:07,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:07,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:07,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. after waiting 0 ms 2023-04-28 04:58:07,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:07,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 04:58:07,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:07,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:07,928 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a6bab1698a9b263a4df9a801810dbac0 to self. 2023-04-28 04:58:07,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:07,932 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=CLOSED 2023-04-28 04:58:07,932 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657887932"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657887932"}]},"ts":"1682657887932"} 2023-04-28 04:58:07,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-28 04:58:07,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 172 msec 2023-04-28 04:58:07,938 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase12.apache.org,37663,1682657733313; forceNewPlan=false, retain=true 2023-04-28 04:58:08,089 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=OPENING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,089 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657888089"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657888089"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657888089"}]},"ts":"1682657888089"} 2023-04-28 04:58:08,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:58:08,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a6bab1698a9b263a4df9a801810dbac0, NAME => 'TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.', STARTKEY => '', ENDKEY => ''} 2023-04-28 04:58:08,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 04:58:08,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7894): checking encryption for a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(7897): checking classloading for a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,250 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,252 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A 2023-04-28 04:58:08,252 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A 2023-04-28 04:58:08,252 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:58:08,252 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6bab1698a9b263a4df9a801810dbac0 columnFamilyName A 2023-04-28 04:58:08,255 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(310): Store=a6bab1698a9b263a4df9a801810dbac0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:58:08,255 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,257 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B 2023-04-28 04:58:08,257 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B 2023-04-28 04:58:08,257 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:58:08,257 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6bab1698a9b263a4df9a801810dbac0 columnFamilyName B 2023-04-28 04:58:08,258 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(310): Store=a6bab1698a9b263a4df9a801810dbac0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:58:08,258 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,260 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C 2023-04-28 04:58:08,260 DEBUG [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C 2023-04-28 04:58:08,260 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 04:58:08,260 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6bab1698a9b263a4df9a801810dbac0 columnFamilyName C 2023-04-28 04:58:08,261 INFO [StoreOpener-a6bab1698a9b263a4df9a801810dbac0-1] regionserver.HStore(310): Store=a6bab1698a9b263a4df9a801810dbac0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 04:58:08,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 04:58:08,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1055): writing seq id for a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1072): Opened a6bab1698a9b263a4df9a801810dbac0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61661375, jitterRate=-0.08117391169071198}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 04:58:08,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(965): Region open journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:08,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., pid=61, masterSystemTime=1682657888242 2023-04-28 04:58:08,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase12:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,274 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,274 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657888274"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682657888274"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682657888274"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682657888274"}]},"ts":"1682657888274"} 2023-04-28 04:58:08,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-28 04:58:08,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 185 msec 2023-04-28 04:58:08,278 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-28 04:58:08,278 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, REOPEN/MOVE in 518 msec 2023-04-28 04:58:08,280 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-28 04:58:08,280 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 523 msec 2023-04-28 04:58:08,282 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 940 msec 2023-04-28 04:58:08,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-28 04:58:08,284 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x2f11e84a to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,401 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec98d63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,402 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x3a1010a6 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,421 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ae3ec16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,421 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x4143b9ae to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,431 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bec0400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,432 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x686af57e to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,454 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b8bf14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,455 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x078cd331 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,473 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@387217bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,474 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x7c00a18c to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,492 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5467be78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,492 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x3da68fe0 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,505 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44535519, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,505 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x7fd90ac8 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,515 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7689435c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,516 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x70e63396 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,544 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@475ee808, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,545 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(139): Connect 0x32cc4808 to 127.0.0.1:52932 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 04:58:08,587 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7377b744, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 04:58:08,644 DEBUG [hconnection-0x2db45121-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,645 DEBUG [hconnection-0x15a9af97-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,649 DEBUG [hconnection-0x507aec67-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,650 DEBUG [hconnection-0x287d5471-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,652 DEBUG [hconnection-0x2aff626f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,652 DEBUG [hconnection-0xeeb913c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,653 DEBUG [hconnection-0x42914c79-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,653 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,653 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,654 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47418, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,655 DEBUG [hconnection-0x72f8b65a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,656 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:08,661 DEBUG [hconnection-0x651d45ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,662 DEBUG [hconnection-0x30e74488-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 04:58:08,663 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,664 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,665 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,665 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47452, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,665 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47440, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,665 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,669 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.104.168:47464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 04:58:08,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:08,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:08,677 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:08,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:08,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:08,677 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:08,678 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,678 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:08,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:58:08,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:08,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:08,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:08,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:08,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:08,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:08,691 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:08,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:08,691 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,692 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,692 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:08,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:08,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657948692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:08,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:08,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657888690, max timestamp=1682657888695Num uniques 859; 2023-04-28 04:58:08,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:08,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:08,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:08,696 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:08,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657888690, max timestamp=1682657888695Num uniques 859; 2023-04-28 04:58:08,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657888690, max timestamp=1682657888695Num uniques 859; 2023-04-28 04:58:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:08,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:08,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:08,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:08,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:08,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657948706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657948707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657948707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657948708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,715 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:08,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a39c2f[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:08,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:08,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:08,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:08,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:08,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:08,736 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:08,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:08,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:08,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:08,737 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:08,738 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:08,738 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:08,738 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:08,738 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:08,776 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,776 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,776 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:08,776 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:08,777 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657948797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:08,799 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:08,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:08,799 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:08,799 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:08,799 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:08,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,799 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:08,799 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657948809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657948810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657948810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:08,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657948810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,900 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:08,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:08,901 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:08,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:08,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:08,903 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:08,903 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:08,903 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:08,903 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:08,903 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,903 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:08,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:08,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:08,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:08,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:08,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:08,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:08,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:08,916 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,916 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:08,916 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:08,916 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:08,916 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,917 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:08,917 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:08,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:08,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:08,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:08,926 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:08,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73b085a9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:08,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,932 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 04:58:08,936 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,936 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:08,936 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:08,936 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:08,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:08,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:08,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:08,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:08,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:08,954 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:08,954 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:08,954 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:08,954 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:08,954 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:08,965 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,965 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:08,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,965 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:08,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:08,965 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,966 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:08,978 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:08,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:08,978 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:08,978 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:08,978 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:08,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,978 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:08,978 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657948999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657949011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657949011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657949011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657949012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,078 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:09,080 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:09,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:09,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:09,081 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:09,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:09,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:09,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,081 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:09,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:09,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:09,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:09,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:09,095 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:09,095 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,095 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:09,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:09,096 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:09,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:09,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:09,096 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:09,097 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289ad7d3d4836c426b8ebfa2ec167b2c3c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ad7d3d4836c426b8ebfa2ec167b2c3c 2023-04-28 04:58:09,104 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,104 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,104 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,104 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:09,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:09,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50421870[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:09,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:09,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:09,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:09,115 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:09,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:09,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:09,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:09,127 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:09,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:09,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:09,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:09,127 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:09,136 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,136 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,136 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,136 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,137 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,146 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:09,146 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:09,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,146 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:09,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:09,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,147 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,247 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:09,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:09,249 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:09,250 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:09,250 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,250 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:09,250 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:09,250 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,250 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:09,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:09,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:09,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:09,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:09,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:09,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:09,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:09,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:09,268 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:09,268 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:09,280 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,280 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,281 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:09,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62afbc36[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:09,282 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:09,282 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:09,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:09,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:09,294 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:09,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657949303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:09,304 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,304 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:09,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:09,306 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:09,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:09,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:09,306 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:09,306 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:09,315 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,315 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,315 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657949315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,316 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657949315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657949316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657949317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,387 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,387 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:09,388 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:09,387 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:09,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,388 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:09,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,388 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:09,488 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:09,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:09,490 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:09,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:09,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:09,492 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:09,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:09,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:09,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,501 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/2e1f1e95885c4db7bb74481d1cbb507a 2023-04-28 04:58:09,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:09,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:09,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:09,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:09,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:09,557 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,557 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:09,557 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:09,557 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:09,557 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:09,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:09,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,578 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:09,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c7306a8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:09,579 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:09,580 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,588 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,588 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,589 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:09,588 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:09,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:09,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:09,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,599 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:09,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:09,601 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:09,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:09,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:09,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:09,601 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:09,609 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,609 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,609 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,610 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:09,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,620 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:09,620 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,620 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:09,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,721 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:09,723 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:09,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:09,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:09,724 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:09,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:09,724 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:09,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,736 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:09,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:09,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:09,737 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:09,737 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,737 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:09,740 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,740 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:09,740 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:09,740 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:09,740 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:09,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,746 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,747 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,747 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:09,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b995942[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:09,749 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:09,749 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,757 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:09,757 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:09,757 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:09,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:09,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,767 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,768 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:09,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:09,771 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:09,771 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:09,771 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:09,771 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:09,771 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:09,778 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,778 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,778 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,778 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,779 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,788 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,788 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,788 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,788 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:09,789 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:09,788 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,788 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,788 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:09,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,789 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,789 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:09,789 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:09,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657949805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657949817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657949817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657949820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657949821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,889 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:09,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:09,891 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:09,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:09,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:09,892 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:09,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:09,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:09,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,892 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,904 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,904 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:09,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:09,904 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,905 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:09,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:09,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:09,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:09,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:09,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:09,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:09,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:09,905 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:09,906 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,906 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:09,906 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:09,906 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:09,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:09,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:09,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:09,906 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:09,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/05426c38bc9d4536be7b3cad3387a873 2023-04-28 04:58:09,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,915 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,915 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,915 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:09,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b5841d9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:09,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:09,918 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:09,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:09,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:09,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:09,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:09,936 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:09,936 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,936 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,936 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:09,937 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:09,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:09,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:09,938 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:09,938 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:09,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:09,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:09,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:09,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,948 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,029 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:10,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:10,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:10,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:10,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:10,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:10,029 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:10,029 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:10,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,030 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:10,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:10,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:10,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,030 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:10,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,130 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:10,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:10,132 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:10,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:10,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:10,134 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:10,134 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:10,134 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:10,134 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:10,135 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,135 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,165 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,165 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:10,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:10,165 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,165 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:10,166 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:10,166 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:10,166 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:10,166 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:10,166 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:10,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:10,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:10,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:10,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54890a88[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:10,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:10,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,189 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:10,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:10,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:10,189 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:10,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,199 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:10,199 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:10,199 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:10,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:10,201 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:10,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:10,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:10,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:10,201 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:10,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:10,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:10,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:10,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,211 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,220 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:10,220 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:10,220 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:10,220 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:10,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:10,220 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,220 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:10,220 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,220 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:10,220 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,220 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:10,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,221 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,321 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:10,322 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:10,323 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:10,323 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:10,323 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:10,323 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:10,324 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,324 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/9f73f635600b476cb43724f7552f712e 2023-04-28 04:58:10,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/2e1f1e95885c4db7bb74481d1cbb507a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/2e1f1e95885c4db7bb74481d1cbb507a 2023-04-28 04:58:10,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/2e1f1e95885c4db7bb74481d1cbb507a, entries=150, sequenceid=16, filesize=25.1 K 2023-04-28 04:58:10,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/05426c38bc9d4536be7b3cad3387a873 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/05426c38bc9d4536be7b3cad3387a873 2023-04-28 04:58:10,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,336 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:10,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:10,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:10,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:10,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:10,339 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:10,339 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:10,339 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:10,339 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:10,339 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:10,339 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:10,340 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:10,340 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:10,340 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:10,340 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:10,340 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:10,340 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,340 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:10,340 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:10,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,347 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/05426c38bc9d4536be7b3cad3387a873, entries=150, sequenceid=16, filesize=11.6 K 2023-04-28 04:58:10,347 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:10,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,347 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:10,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/9f73f635600b476cb43724f7552f712e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9f73f635600b476cb43724f7552f712e 2023-04-28 04:58:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:10,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:10,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@302b10f7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:10,351 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:10,351 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9f73f635600b476cb43724f7552f712e, entries=150, sequenceid=16, filesize=11.6 K 2023-04-28 04:58:10,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for a6bab1698a9b263a4df9a801810dbac0 in 1673ms, sequenceid=16, compaction requested=false 2023-04-28 04:58:10,352 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 04:58:10,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:10,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:10,363 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:10,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:10,364 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:10,375 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,375 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:10,375 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:10,376 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:10,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:10,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:10,378 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:10,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:10,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:10,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:10,379 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:10,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:10,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,389 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,391 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,399 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:10,399 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:10,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:10,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,399 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,399 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:10,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:10,400 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:10,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:10,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:10,499 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:10,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:10,503 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:10,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:10,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:10,520 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:10,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:10,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:10,520 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:10,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,522 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,533 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,533 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:10,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,534 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,534 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:10,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:10,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:10,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:10,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:10,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:10,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:10,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:10,537 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:10,538 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:10,538 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:10,538 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:10,538 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:10,538 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-28 04:58:10,538 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:10,538 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:10,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:10,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:10,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:10,539 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:10,583 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c4808a34e4b74ab4aeb945d8e113f357 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c4808a34e4b74ab4aeb945d8e113f357 2023-04-28 04:58:10,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:10,811 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:10,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657950824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657950825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657950826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657950827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657950831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657950928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657950928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:10,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657950929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:10,987 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/ec572e3092624ff097b186e52d55931b 2023-04-28 04:58:11,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657951130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657951130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657951131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,409 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/951d968c493c4cd7a2114103ef5bb707 2023-04-28 04:58:11,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657951433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657951434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657951434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,818 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a098cfa715144c25a96001777ca3fac0 2023-04-28 04:58:11,822 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/ec572e3092624ff097b186e52d55931b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/ec572e3092624ff097b186e52d55931b 2023-04-28 04:58:11,837 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/ec572e3092624ff097b186e52d55931b, entries=150, sequenceid=50, filesize=25.1 K 2023-04-28 04:58:11,838 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/951d968c493c4cd7a2114103ef5bb707 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/951d968c493c4cd7a2114103ef5bb707 2023-04-28 04:58:11,841 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/951d968c493c4cd7a2114103ef5bb707, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 04:58:11,842 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a098cfa715144c25a96001777ca3fac0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a098cfa715144c25a96001777ca3fac0 2023-04-28 04:58:11,846 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a098cfa715144c25a96001777ca3fac0, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 04:58:11,847 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for a6bab1698a9b263a4df9a801810dbac0 in 1309ms, sequenceid=50, compaction requested=false 2023-04-28 04:58:11,848 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:11,848 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:11,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:11,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:11,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:11,849 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:11,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:11,862 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:11,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:11,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:11,865 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:11,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b90cf9[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:11,865 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,873 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:11,873 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:11,873 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:11,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:11,883 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:11,884 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:11,884 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:11,884 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:11,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:11,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:11,887 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:11,887 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:11,887 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:11,887 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:11,887 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:11,894 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,894 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:11,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,894 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:11,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:11,895 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:11,898 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:11,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:11,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:11,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:11,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:11,917 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:11,917 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:11,917 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:11,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:11,917 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:11,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:11,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:58:11,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:11,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:11,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:11,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:11,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:11,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657951974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657951974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657951976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,017 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:12,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:12,019 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:12,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:12,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:12,020 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:12,020 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,020 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:12,020 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:12,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,021 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,057 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,057 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,057 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,057 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:12,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:12,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:12,058 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:12,058 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,058 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:12,058 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:12,058 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:12,058 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,058 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:12,058 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:12,058 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:12,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:12,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:12,059 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:12,080 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,080 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,081 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,081 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:12,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657952078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657952078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657952078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:12,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:12,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7866cd9e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:12,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,094 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,094 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:12,094 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:12,094 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:12,104 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,104 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,104 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,104 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:12,106 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:12,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:12,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:12,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:12,106 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:12,106 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:12,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,115 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,116 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,125 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:12,125 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:12,125 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:12,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,126 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:12,126 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,126 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:12,226 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:12,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:12,227 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:12,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:12,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:12,228 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:12,228 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,228 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:12,228 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:12,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,229 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,241 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:12,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:12,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:12,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:12,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:12,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:12,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:12,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:12,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:12,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:12,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:12,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:12,243 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:12,251 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,251 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:12,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64275550[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:12,253 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:12,253 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,262 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:12,262 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:12,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:12,262 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:12,272 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:12,273 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,273 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:12,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:12,274 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:12,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:12,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:12,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:12,274 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:12,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657952282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657952282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,284 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657952283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,293 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,293 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,293 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,294 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:12,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,294 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:12,294 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:12,294 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,294 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:12,294 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:12,359 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280c7bc51c6f6641d09b1e323686fe486b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c7bc51c6f6641d09b1e323686fe486b 2023-04-28 04:58:12,394 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:12,396 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:12,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:12,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:12,397 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:12,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:12,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:12,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,397 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,409 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,409 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,410 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,410 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:12,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:12,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:12,411 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,411 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:12,411 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:12,411 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:12,411 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:12,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,420 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,420 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,420 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:12,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce34d8b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:12,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:12,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,431 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:12,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:12,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:12,432 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:12,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:12,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,441 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:12,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:12,443 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:12,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:12,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:12,443 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:12,443 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:12,451 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,451 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,451 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,452 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,454 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:12,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,462 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:12,462 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,462 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:12,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,462 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:12,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,463 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,563 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:12,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:12,565 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:12,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:12,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:12,568 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:12,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:12,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:12,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,568 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,578 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:12,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:12,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:12,579 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,579 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:12,579 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:12,579 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:12,579 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:12,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657952584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657952586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,588 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,588 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,589 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657952587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:12,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3504462a[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:12,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:12,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,660 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,660 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:12,660 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:12,661 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:12,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:12,683 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:12,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:12,686 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:12,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:12,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:12,686 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:12,687 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:12,694 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,694 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,694 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,694 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,704 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:12,704 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:12,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:12,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,705 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:12,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,705 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:12,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,763 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/fe52281201c0435c90072c196f7206c1 2023-04-28 04:58:12,805 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:12,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:12,807 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:12,808 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:12,808 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,808 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:12,808 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:12,808 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,809 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,820 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:12,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:12,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:12,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:12,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:12,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:12,821 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:12,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,821 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:12,822 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:12,822 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:12,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:12,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:12,822 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:12,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,833 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:12,834 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:12,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b406729[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:12,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:12,836 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,841 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,841 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:12,841 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:12,841 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:12,841 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,841 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657952845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,849 DEBUG [Thread-3192] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:58:12,851 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,851 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:12,851 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,852 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657952850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,853 DEBUG [Thread-3200] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:58:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:12,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:12,855 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:12,855 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:12,855 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:12,855 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:12,855 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:12,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:12,864 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:12,865 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:12,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,873 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:12,873 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:12,873 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:12,873 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:12,873 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:12,973 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:12,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:12,975 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:12,980 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:12,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:12,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:12,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,980 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,988 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,988 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:12,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:12,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,989 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:12,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:12,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:12,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:12,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:12,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:12,990 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:12,991 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:12,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:12,992 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,992 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:12,992 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:12,992 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:12,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:12,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:12,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:12,992 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:12,999 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:12,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:12,999 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:13,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251320c4[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:13,000 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:13,000 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,009 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,010 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:13,010 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:13,010 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:13,010 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,010 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:13,020 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:13,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:13,022 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:13,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:13,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:13,022 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:13,022 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:13,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,031 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,033 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,041 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:13,041 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,041 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,041 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:13,042 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:13,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:13,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657953089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:13,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657953090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:13,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657953094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,142 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:13,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:13,143 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:13,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:13,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:13,144 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:13,144 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,144 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:13,144 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:13,145 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,145 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,157 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,157 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:13,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:13,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:13,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:13,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,158 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:13,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:13,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:13,160 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:13,160 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:13,167 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:13,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330abef8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:13,170 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:13,171 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/241457af47d3401c97bc5a3e2724511c 2023-04-28 04:58:13,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:13,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:13,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:13,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:13,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,239 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,239 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:13,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:13,241 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:13,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:13,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:13,241 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:13,241 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:13,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,273 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,273 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:13,274 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:13,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,273 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,274 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:13,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,274 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,274 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:13,374 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:13,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:13,376 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:13,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:13,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:13,377 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:13,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:13,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:13,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,377 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,389 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:13,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:13,390 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,390 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:13,390 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:13,390 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:13,390 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:13,399 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,399 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,399 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,399 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:13,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:13,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a88ba5[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:13,400 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,409 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,409 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:13,409 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:13,409 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,409 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:13,409 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:13,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,420 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:13,420 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,420 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:13,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:13,422 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:13,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:13,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:13,422 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:13,422 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:13,430 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,430 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,430 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,431 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,432 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,441 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,441 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:13,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:13,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,441 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:13,442 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:13,442 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:13,542 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:13,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:13,545 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:13,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:13,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:13,546 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:13,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:13,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:13,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,557 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,557 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,557 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,557 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:13,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:13,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:13,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:13,558 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:13,559 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,559 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:13,559 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:13,559 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,559 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:13,560 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:13,567 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,567 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,568 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,568 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c29d936[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:13,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:13,569 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,578 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,578 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:13,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:13,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:13,578 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:13,588 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,588 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:13,588 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,588 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:13,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:13,590 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:13,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:13,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:13,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:13,590 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:13,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:13,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:13,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,601 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:13,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/1e7d15b30f134effa0fc8363ea85f968 2023-04-28 04:58:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/fe52281201c0435c90072c196f7206c1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe52281201c0435c90072c196f7206c1 2023-04-28 04:58:13,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:13,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:13,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe52281201c0435c90072c196f7206c1, entries=150, sequenceid=64, filesize=25.1 K 2023-04-28 04:58:13,616 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:13,616 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:13,616 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:13,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:13,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,616 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:13,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,616 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:13,617 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,616 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:13,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/241457af47d3401c97bc5a3e2724511c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/241457af47d3401c97bc5a3e2724511c 2023-04-28 04:58:13,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/241457af47d3401c97bc5a3e2724511c, entries=150, sequenceid=64, filesize=11.6 K 2023-04-28 04:58:13,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/1e7d15b30f134effa0fc8363ea85f968 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/1e7d15b30f134effa0fc8363ea85f968 2023-04-28 04:58:13,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/1e7d15b30f134effa0fc8363ea85f968, entries=150, sequenceid=64, filesize=11.6 K 2023-04-28 04:58:13,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for a6bab1698a9b263a4df9a801810dbac0 in 1688ms, sequenceid=64, compaction requested=true 2023-04-28 04:58:13,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:13,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:13,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:13,629 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:13,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:13,629 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:13,633 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:13,633 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:13,633 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/A is initiating minor compaction (all files) 2023-04-28 04:58:13,633 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/A in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,633 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/2e1f1e95885c4db7bb74481d1cbb507a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/ec572e3092624ff097b186e52d55931b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe52281201c0435c90072c196f7206c1] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=75.4 K 2023-04-28 04:58:13,633 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/B is initiating minor compaction (all files) 2023-04-28 04:58:13,635 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/B in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,635 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 2e1f1e95885c4db7bb74481d1cbb507a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682657888674 2023-04-28 04:58:13,635 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/05426c38bc9d4536be7b3cad3387a873, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/951d968c493c4cd7a2114103ef5bb707, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/241457af47d3401c97bc5a3e2724511c] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=34.8 K 2023-04-28 04:58:13,636 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting ec572e3092624ff097b186e52d55931b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657888706 2023-04-28 04:58:13,636 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 05426c38bc9d4536be7b3cad3387a873, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682657888674 2023-04-28 04:58:13,636 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting fe52281201c0435c90072c196f7206c1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682657890822 2023-04-28 04:58:13,636 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 951d968c493c4cd7a2114103ef5bb707, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657888706 2023-04-28 04:58:13,637 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 241457af47d3401c97bc5a3e2724511c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682657890822 2023-04-28 04:58:13,644 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#B#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:13,647 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#A#compaction#434 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:13,665 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/b7cc1230acc0497fa438abdaceeb2f6d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b7cc1230acc0497fa438abdaceeb2f6d 2023-04-28 04:58:13,670 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/B of a6bab1698a9b263a4df9a801810dbac0 into b7cc1230acc0497fa438abdaceeb2f6d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:13,670 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:13,670 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/B, priority=13, startTime=1682657893629; duration=0sec 2023-04-28 04:58:13,670 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:13,670 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:13,671 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:13,671 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/C is initiating minor compaction (all files) 2023-04-28 04:58:13,671 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/C in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,671 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9f73f635600b476cb43724f7552f712e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a098cfa715144c25a96001777ca3fac0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/1e7d15b30f134effa0fc8363ea85f968] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=34.8 K 2023-04-28 04:58:13,672 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9f73f635600b476cb43724f7552f712e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682657888674 2023-04-28 04:58:13,672 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a098cfa715144c25a96001777ca3fac0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682657888706 2023-04-28 04:58:13,672 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 1e7d15b30f134effa0fc8363ea85f968, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682657890822 2023-04-28 04:58:13,679 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#C#compaction#436 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:13,684 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/5ba4dd76f97c49758f0ee773dfd53774 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5ba4dd76f97c49758f0ee773dfd53774 2023-04-28 04:58:13,688 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/C of a6bab1698a9b263a4df9a801810dbac0 into 5ba4dd76f97c49758f0ee773dfd53774(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:13,688 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:13,688 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/C, priority=13, startTime=1682657893629; duration=0sec 2023-04-28 04:58:13,688 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:13,717 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:13,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:13,718 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:13,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:13,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:13,719 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:13,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:13,719 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:13,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,720 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:13,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:13,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:13,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:13,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:13,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:13,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:13,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:13,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:13,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:13,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:13,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:13,726 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:13,727 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:13,727 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:14,080 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/eba74652b574409db185f53527ff52f6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/eba74652b574409db185f53527ff52f6 2023-04-28 04:58:14,088 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/A of a6bab1698a9b263a4df9a801810dbac0 into eba74652b574409db185f53527ff52f6(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:14,088 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:14,088 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/A, priority=13, startTime=1682657893629; duration=0sec 2023-04-28 04:58:14,089 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:14,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:14,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:14,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657954106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657954107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657954108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,142 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c9ab39717f6a47d4b2ca8862002cada1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c9ab39717f6a47d4b2ca8862002cada1 2023-04-28 04:58:14,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657954209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657954209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657954210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657954411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657954412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657954413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,549 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/348c49e51d3443ed9ac5eac0e777c98d 2023-04-28 04:58:14,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657954713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657954715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:14,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657954717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:14,965 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/55d66d95370d4d1a8dedde7cc1e9d1d5 2023-04-28 04:58:15,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:15,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657955218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:15,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657955219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:15,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657955219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,374 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/c711fa60ec954051b65e0f97e149c6f2 2023-04-28 04:58:15,376 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/348c49e51d3443ed9ac5eac0e777c98d as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/348c49e51d3443ed9ac5eac0e777c98d 2023-04-28 04:58:15,379 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/348c49e51d3443ed9ac5eac0e777c98d, entries=150, sequenceid=85, filesize=25.1 K 2023-04-28 04:58:15,380 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/55d66d95370d4d1a8dedde7cc1e9d1d5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/55d66d95370d4d1a8dedde7cc1e9d1d5 2023-04-28 04:58:15,382 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/55d66d95370d4d1a8dedde7cc1e9d1d5, entries=150, sequenceid=85, filesize=11.6 K 2023-04-28 04:58:15,383 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/c711fa60ec954051b65e0f97e149c6f2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c711fa60ec954051b65e0f97e149c6f2 2023-04-28 04:58:15,385 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c711fa60ec954051b65e0f97e149c6f2, entries=150, sequenceid=85, filesize=11.6 K 2023-04-28 04:58:15,386 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a6bab1698a9b263a4df9a801810dbac0 in 1659ms, sequenceid=85, compaction requested=false 2023-04-28 04:58:15,386 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:15,386 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:15,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:15,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:15,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:15,386 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:15,448 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,448 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:15,449 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,449 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6080d777[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:15,450 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:15,450 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,462 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:15,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:15,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:15,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:15,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:15,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:15,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:15,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:15,474 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:15,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:15,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:15,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:15,475 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:15,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:15,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:15,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:15,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:15,524 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:15,524 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:15,524 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:15,524 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:15,524 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:15,524 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,624 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:15,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:15,626 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:15,627 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:15,627 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:15,627 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:15,627 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:15,627 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,627 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:15,648 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:15,649 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:15,649 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:15,649 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:15,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:15,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:15,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:15,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:15,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:15,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:15,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:15,650 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:15,650 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:15,650 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:16,062 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042879cf396bbe664fbeb659121ff05a13b3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042879cf396bbe664fbeb659121ff05a13b3 2023-04-28 04:58:16,067 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/e5e0944e689b40eda75a9469e0a8b0f8 2023-04-28 04:58:16,076 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/89d517f02ffc426e857a5a3e43f9cdd9 2023-04-28 04:58:16,084 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/74daae97538e4d6391e93c9f7db14bef 2023-04-28 04:58:16,087 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/e5e0944e689b40eda75a9469e0a8b0f8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/e5e0944e689b40eda75a9469e0a8b0f8 2023-04-28 04:58:16,089 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/e5e0944e689b40eda75a9469e0a8b0f8, entries=150, sequenceid=99, filesize=25.1 K 2023-04-28 04:58:16,090 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/89d517f02ffc426e857a5a3e43f9cdd9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/89d517f02ffc426e857a5a3e43f9cdd9 2023-04-28 04:58:16,092 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/89d517f02ffc426e857a5a3e43f9cdd9, entries=150, sequenceid=99, filesize=11.6 K 2023-04-28 04:58:16,093 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/74daae97538e4d6391e93c9f7db14bef as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/74daae97538e4d6391e93c9f7db14bef 2023-04-28 04:58:16,102 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/74daae97538e4d6391e93c9f7db14bef, entries=150, sequenceid=99, filesize=11.6 K 2023-04-28 04:58:16,103 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for a6bab1698a9b263a4df9a801810dbac0 in 453ms, sequenceid=99, compaction requested=true 2023-04-28 04:58:16,103 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:16,103 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,103 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:16,103 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:16,103 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,103 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:16,103 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:16,154 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,154 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,154 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,154 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:16,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b58e1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:16,156 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:16,156 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,167 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:16,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:16,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:16,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:16,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:16,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,178 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:16,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:16,180 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:16,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:16,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:16,180 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:16,180 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:16,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,189 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,190 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,199 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:16,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,199 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:16,199 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:16,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:58:16,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:16,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:16,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:16,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:16,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:16,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:16,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657956279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657956283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657956283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,299 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:16,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:16,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:16,302 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:16,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:16,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:16,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,314 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:16,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:16,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:16,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:16,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:16,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:16,325 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,325 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,325 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,325 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@660e1e34[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:16,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:16,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,335 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,336 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:16,336 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:16,336 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:16,336 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:16,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:16,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:16,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:16,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:16,348 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:16,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:16,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:16,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:16,348 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:16,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,357 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,367 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:16,367 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:16,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:16,367 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,367 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:16,368 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,368 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:16,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,368 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,368 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,368 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657956384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657956387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657956388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,468 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:16,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:16,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:16,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:16,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:16,470 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:16,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,470 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:16,471 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:16,471 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,471 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:16,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:16,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:16,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:16,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:16,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:16,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:16,484 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:16,484 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,484 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:16,485 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:16,485 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:16,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:16,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:16,485 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:16,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,493 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,494 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,494 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:16,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747c8baa[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:16,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:16,495 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,504 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,504 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:16,504 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:16,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:16,504 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:16,514 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,514 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:16,514 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,514 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:16,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:16,516 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:16,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:16,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:16,517 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:16,517 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:16,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,526 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,526 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,526 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,565 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,565 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,565 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:16,565 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:16,565 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,565 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,566 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,566 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,566 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,566 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,565 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,566 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,566 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:16,566 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657956588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657956591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657956591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,666 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:16,667 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428dc52512e932c4548b9dc5cbdc5397ca7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dc52512e932c4548b9dc5cbdc5397ca7 2023-04-28 04:58:16,668 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:16,669 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:16,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:16,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:16,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,672 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/c2106e2ed4784e30b196f313002b0c85 2023-04-28 04:58:16,693 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,693 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,694 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,694 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:16,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:16,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:16,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:16,695 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:16,695 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,695 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:16,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:16,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:16,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:16,704 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:16,704 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:16,714 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,714 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,715 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:16,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1221d13d[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:16,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:16,716 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:16,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:16,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:16,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:16,736 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,737 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:16,738 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,738 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:16,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:16,739 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:16,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:16,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:16,739 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:16,739 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:16,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,747 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,757 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:16,757 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,757 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,757 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:16,857 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:16,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:16,859 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:16,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47380 deadline: 1682657956858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,860 DEBUG [Thread-3192] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:58:16,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:16,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:16,861 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:16,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:16,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:16,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,861 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,872 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,872 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,873 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:16,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:16,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:16,873 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:16,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:16,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:16,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:16,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:16,874 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47464 deadline: 1682657956872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,874 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:16,874 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:16,874 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:16,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:16,874 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:16,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:16,875 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:16,875 DEBUG [Thread-3200] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., hostname=jenkins-hbase12.apache.org,37663,1682657733313, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 04:58:16,883 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,883 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,883 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,883 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:16,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e21dfc8[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:16,884 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:16,884 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,893 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,894 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:16,894 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:16,894 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:16,894 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,894 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:16,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657956892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657956894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:16,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657956894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,904 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:16,904 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,904 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,904 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:16,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:16,906 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:16,906 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:16,906 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:16,906 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:16,906 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:16,914 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:16,914 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,915 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:16,915 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,916 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:16,925 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:16,925 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:16,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:16,925 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,925 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:16,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:16,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,026 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:17,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:17,027 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:17,029 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:17,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:17,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:17,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:17,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,029 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,041 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,041 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:17,041 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,041 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:17,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:17,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:17,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:17,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:17,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:17,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:17,044 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:17,044 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:17,044 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:17,045 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:17,045 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:17,045 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:17,045 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:17,045 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,045 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:17,045 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:17,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,051 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:17,052 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,052 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:17,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:17,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58489786[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:17,053 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:17,053 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,062 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,062 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:17,062 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:17,062 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:17,062 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,062 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:17,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:17,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:17,072 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:17,074 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:17,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:17,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:17,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:17,074 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:17,083 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,083 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:17,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,083 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:17,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:17,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:17,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/64ba3a702bc94e889c67376544a14179 2023-04-28 04:58:17,084 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/9872699a3aaa41219e5f3e75cd8d57c8 2023-04-28 04:58:17,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,093 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:17,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:17,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:17,093 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:17,094 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:17,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:17,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:17,094 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:17,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:17,094 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,094 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:17,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:17,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/c2106e2ed4784e30b196f313002b0c85 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/c2106e2ed4784e30b196f313002b0c85 2023-04-28 04:58:17,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/c2106e2ed4784e30b196f313002b0c85, entries=250, sequenceid=111, filesize=38.7 K 2023-04-28 04:58:17,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/64ba3a702bc94e889c67376544a14179 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/64ba3a702bc94e889c67376544a14179 2023-04-28 04:58:17,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/64ba3a702bc94e889c67376544a14179, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:58:17,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/9872699a3aaa41219e5f3e75cd8d57c8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9872699a3aaa41219e5f3e75cd8d57c8 2023-04-28 04:58:17,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9872699a3aaa41219e5f3e75cd8d57c8, entries=150, sequenceid=111, filesize=11.6 K 2023-04-28 04:58:17,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a6bab1698a9b263a4df9a801810dbac0 in 854ms, sequenceid=111, compaction requested=true 2023-04-28 04:58:17,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:17,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:17,105 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:58:17,105 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:58:17,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:17,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:17,105 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 116971 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:58:17,105 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:58:17,105 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/A is initiating minor compaction (all files) 2023-04-28 04:58:17,106 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/B is initiating minor compaction (all files) 2023-04-28 04:58:17,106 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/A in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:17,106 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/B in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:17,106 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/eba74652b574409db185f53527ff52f6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/348c49e51d3443ed9ac5eac0e777c98d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/e5e0944e689b40eda75a9469e0a8b0f8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/c2106e2ed4784e30b196f313002b0c85] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=114.2 K 2023-04-28 04:58:17,106 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b7cc1230acc0497fa438abdaceeb2f6d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/55d66d95370d4d1a8dedde7cc1e9d1d5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/89d517f02ffc426e857a5a3e43f9cdd9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/64ba3a702bc94e889c67376544a14179] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=46.5 K 2023-04-28 04:58:17,106 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting eba74652b574409db185f53527ff52f6, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682657890822 2023-04-28 04:58:17,106 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b7cc1230acc0497fa438abdaceeb2f6d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682657890822 2023-04-28 04:58:17,106 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 348c49e51d3443ed9ac5eac0e777c98d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682657891960 2023-04-28 04:58:17,106 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 55d66d95370d4d1a8dedde7cc1e9d1d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682657891960 2023-04-28 04:58:17,107 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 89d517f02ffc426e857a5a3e43f9cdd9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682657894104 2023-04-28 04:58:17,107 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting e5e0944e689b40eda75a9469e0a8b0f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682657894104 2023-04-28 04:58:17,107 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 64ba3a702bc94e889c67376544a14179, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657896246 2023-04-28 04:58:17,107 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting c2106e2ed4784e30b196f313002b0c85, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657896238 2023-04-28 04:58:17,112 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#B#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:17,115 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#A#compaction#446 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:17,118 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/d2a540177603425e819b93cfda601478 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/d2a540177603425e819b93cfda601478 2023-04-28 04:58:17,123 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/B of a6bab1698a9b263a4df9a801810dbac0 into d2a540177603425e819b93cfda601478(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:17,123 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:17,123 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/B, priority=12, startTime=1682657897105; duration=0sec 2023-04-28 04:58:17,124 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:17,124 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:58:17,124 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:58:17,125 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/C is initiating minor compaction (all files) 2023-04-28 04:58:17,125 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/C in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:17,125 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5ba4dd76f97c49758f0ee773dfd53774, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c711fa60ec954051b65e0f97e149c6f2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/74daae97538e4d6391e93c9f7db14bef, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9872699a3aaa41219e5f3e75cd8d57c8] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=46.5 K 2023-04-28 04:58:17,125 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5ba4dd76f97c49758f0ee773dfd53774, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682657890822 2023-04-28 04:58:17,125 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c711fa60ec954051b65e0f97e149c6f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682657891960 2023-04-28 04:58:17,126 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 74daae97538e4d6391e93c9f7db14bef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682657894104 2023-04-28 04:58:17,126 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9872699a3aaa41219e5f3e75cd8d57c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657896246 2023-04-28 04:58:17,136 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#C#compaction#448 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:17,138 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/4da84585b32244a5af85281029e03066 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/4da84585b32244a5af85281029e03066 2023-04-28 04:58:17,142 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/A of a6bab1698a9b263a4df9a801810dbac0 into 4da84585b32244a5af85281029e03066(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:17,142 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:17,142 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/A, priority=12, startTime=1682657897104; duration=0sec 2023-04-28 04:58:17,142 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:17,146 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/03a4e99560544c299a883d7931d618e3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/03a4e99560544c299a883d7931d618e3 2023-04-28 04:58:17,149 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/C of a6bab1698a9b263a4df9a801810dbac0 into 03a4e99560544c299a883d7931d618e3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:17,149 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:17,149 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/C, priority=12, startTime=1682657897105; duration=0sec 2023-04-28 04:58:17,149 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:17,194 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:17,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:17,196 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:17,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:17,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:17,197 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:17,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:17,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:17,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:17,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,197 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,240 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:17,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:17,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,240 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:17,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:17,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:17,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:17,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:17,241 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:17,241 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:17,241 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:17,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:17,242 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:17,242 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:17,242 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:17,243 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:17,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:17,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:17,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657957402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657957403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657957403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657957504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657957505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657957505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,658 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304286f8740845bb846d89e273db7ff78a586 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286f8740845bb846d89e273db7ff78a586 2023-04-28 04:58:17,673 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/08e87db2611a419da023f9d0766886cd 2023-04-28 04:58:17,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657957706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657957707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:17,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:17,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657957707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657958011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657958011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657958012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,098 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/b86d91e4393d455995d6eb910321e8f3 2023-04-28 04:58:18,149 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/5302371a9bfc417f8ee513188d193e66 2023-04-28 04:58:18,152 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/08e87db2611a419da023f9d0766886cd as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/08e87db2611a419da023f9d0766886cd 2023-04-28 04:58:18,156 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/08e87db2611a419da023f9d0766886cd, entries=150, sequenceid=134, filesize=25.2 K 2023-04-28 04:58:18,156 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/b86d91e4393d455995d6eb910321e8f3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b86d91e4393d455995d6eb910321e8f3 2023-04-28 04:58:18,160 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b86d91e4393d455995d6eb910321e8f3, entries=150, sequenceid=134, filesize=11.7 K 2023-04-28 04:58:18,161 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/5302371a9bfc417f8ee513188d193e66 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5302371a9bfc417f8ee513188d193e66 2023-04-28 04:58:18,166 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5302371a9bfc417f8ee513188d193e66, entries=150, sequenceid=134, filesize=11.7 K 2023-04-28 04:58:18,168 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a6bab1698a9b263a4df9a801810dbac0 in 926ms, sequenceid=134, compaction requested=false 2023-04-28 04:58:18,168 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:18,168 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:18,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:18,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:18,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:18,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:18,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,238 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:18,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,238 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:18,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:18,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f976fe[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:18,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:18,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,251 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:18,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:18,252 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:18,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,252 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:18,262 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:18,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:18,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:18,264 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:18,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:18,279 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:18,279 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:18,279 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:18,279 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:18,279 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:18,291 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,291 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:18,291 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:18,291 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,296 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,304 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:18,304 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:18,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:18,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:18,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:18,305 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:18,305 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:18,304 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:18,304 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:18,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:18,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:18,305 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:18,405 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:18,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:18,409 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:18,414 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:18,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:18,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:18,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:18,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,414 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:18,420 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:18,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:18,420 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:18,420 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,421 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:18,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:18,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:18,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:18,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:18,424 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:18,424 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:18,424 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:18,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:18,437 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:18,437 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:18,437 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:18,437 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 04:58:18,437 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:18,437 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:18,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:18,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:18,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:18,438 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:18,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:18,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657898564, max timestamp=1682657898580Num uniques 859; 2023-04-28 04:58:18,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:58:18,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682657898564, max timestamp=1682657898580Num uniques 859; 2023-04-28 04:58:18,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:58:18,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682657898564, max timestamp=1682657898580Num uniques 859; 2023-04-28 04:58:18,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 04:58:18,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657958601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657958606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657958606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657958707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657958711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657958711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,877 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428df6d731c2611407f87c887ce01109a74 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428df6d731c2611407f87c887ce01109a74 2023-04-28 04:58:18,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657958910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657958914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:18,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:18,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657958914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657959213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657959216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657959218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,281 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/11515da1bdc7421395522dde6b85be9a 2023-04-28 04:58:19,690 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/ebf9cedb71534e958d0de90e8b6288b9 2023-04-28 04:58:19,700 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/e356bc5a5f08421da7b443777b4c0c5e 2023-04-28 04:58:19,703 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/11515da1bdc7421395522dde6b85be9a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/11515da1bdc7421395522dde6b85be9a 2023-04-28 04:58:19,707 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/11515da1bdc7421395522dde6b85be9a, entries=100, sequenceid=146, filesize=18.5 K 2023-04-28 04:58:19,707 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/ebf9cedb71534e958d0de90e8b6288b9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/ebf9cedb71534e958d0de90e8b6288b9 2023-04-28 04:58:19,710 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/ebf9cedb71534e958d0de90e8b6288b9, entries=100, sequenceid=146, filesize=9.4 K 2023-04-28 04:58:19,711 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/e356bc5a5f08421da7b443777b4c0c5e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/e356bc5a5f08421da7b443777b4c0c5e 2023-04-28 04:58:19,715 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/e356bc5a5f08421da7b443777b4c0c5e, entries=100, sequenceid=146, filesize=9.4 K 2023-04-28 04:58:19,716 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for a6bab1698a9b263a4df9a801810dbac0 in 1279ms, sequenceid=146, compaction requested=true 2023-04-28 04:58:19,716 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:19,716 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:19,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:19,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:19,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:19,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:19,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:19,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 04:58:19,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:19,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:19,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:19,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:19,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:19,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:19,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,725 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:19,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,726 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:19,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:19,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544deee5[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:19,727 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:19,727 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657959729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657959730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,732 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428dcb6499fcfd54e31a5c395bdeb4cc52f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dcb6499fcfd54e31a5c395bdeb4cc52f 2023-04-28 04:58:19,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657959732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,735 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:19,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:19,736 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:19,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,736 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:19,746 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:19,746 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:19,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,746 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:19,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:19,748 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:19,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:19,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:19,748 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:19,748 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:19,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:19,756 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,757 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:19,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:19,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:19,757 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,758 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,767 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,767 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:19,767 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:19,767 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:19,767 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:19,767 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:19,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657959833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657959833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:19,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657959835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,867 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:19,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:19,869 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:19,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:19,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:19,871 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:19,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:19,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:19,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:19,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,871 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,883 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,883 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:19,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:19,883 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,883 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:19,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:19,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:19,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:19,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:19,885 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:19,900 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:19,900 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:19,900 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:19,900 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:19,900 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:19,900 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:19,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:19,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:19,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:19,901 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:19,911 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,911 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:19,912 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,912 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:19,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:19,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d2a43b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:19,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:19,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:19,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,925 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:19,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:19,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:19,935 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:19,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:19,935 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:19,937 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:19,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:19,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:19,937 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:19,937 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:19,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:19,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:19,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:19,946 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,947 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,956 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,956 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:19,956 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:19,957 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:19,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:19,956 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:19,956 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,957 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:19,957 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:19,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:19,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:19,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:19,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:19,957 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657960037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657960037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657960038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,057 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:20,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:20,058 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:20,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:20,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:20,060 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:20,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:20,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:20,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,060 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,072 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,072 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,072 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,073 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:20,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:20,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:20,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:20,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:20,073 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:20,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:20,074 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,074 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:20,074 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:20,074 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:20,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:20,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:20,074 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:20,136 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/05ad320028844df2b415b185fdfd8696 2023-04-28 04:58:20,147 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,147 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,148 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8baf580[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:20,149 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:20,149 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,167 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:20,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:20,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:20,167 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:20,177 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:20,177 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:20,177 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:20,177 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:20,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:20,179 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:20,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:20,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:20,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:20,179 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:20,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,189 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,189 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,198 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,198 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,199 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:20,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:20,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:20,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,199 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:20,199 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,199 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:20,299 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:20,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:20,301 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:20,302 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:20,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:20,302 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:20,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,303 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,314 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,314 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,315 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:20,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:20,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:20,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:20,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:20,316 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:20,316 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:20,325 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,325 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,325 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,325 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b772cef[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:20,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:20,326 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,335 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,335 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,335 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:20,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,336 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:20,336 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:20,336 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:20,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47410 deadline: 1682657960340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657960341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47440 deadline: 1682657960343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:20,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:20,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:20,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:20,348 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:20,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:20,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:20,348 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:20,348 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:20,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,357 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,358 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,367 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:20,367 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:20,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,367 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:20,367 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,367 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:20,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:20,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:20,468 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:20,470 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:20,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:20,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:20,477 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:20,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:20,477 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:20,478 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,478 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,486 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,487 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,487 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,487 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:20,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:20,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:20,488 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,488 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:20,488 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:20,488 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:20,488 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:20,489 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:20,501 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,501 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3076d1a2[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:20,503 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:20,503 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/9c2d1681e1054c8c856f74110082b3a1 2023-04-28 04:58:20,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/5f7df0822f36431c97d7da32fb83e46c 2023-04-28 04:58:20,554 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,554 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:20,554 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:20,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,554 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:20,554 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:20,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/05ad320028844df2b415b185fdfd8696 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/05ad320028844df2b415b185fdfd8696 2023-04-28 04:58:20,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/05ad320028844df2b415b185fdfd8696, entries=200, sequenceid=181, filesize=32.1 K 2023-04-28 04:58:20,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/9c2d1681e1054c8c856f74110082b3a1 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/9c2d1681e1054c8c856f74110082b3a1 2023-04-28 04:58:20,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/9c2d1681e1054c8c856f74110082b3a1, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 04:58:20,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/5f7df0822f36431c97d7da32fb83e46c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5f7df0822f36431c97d7da32fb83e46c 2023-04-28 04:58:20,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5f7df0822f36431c97d7da32fb83e46c, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 04:58:20,567 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:20,567 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:20,567 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,567 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:20,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for a6bab1698a9b263a4df9a801810dbac0 in 846ms, sequenceid=181, compaction requested=true 2023-04-28 04:58:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:20,567 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:58:20,567 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:58:20,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:20,568 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,569 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45790 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:58:20,569 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/B is initiating minor compaction (all files) 2023-04-28 04:58:20,569 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/B in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,569 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/d2a540177603425e819b93cfda601478, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b86d91e4393d455995d6eb910321e8f3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/ebf9cedb71534e958d0de90e8b6288b9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/9c2d1681e1054c8c856f74110082b3a1] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=44.7 K 2023-04-28 04:58:20,570 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103616 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:58:20,570 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d2a540177603425e819b93cfda601478, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657896246 2023-04-28 04:58:20,570 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/A is initiating minor compaction (all files) 2023-04-28 04:58:20,570 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/A in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,570 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/4da84585b32244a5af85281029e03066, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/08e87db2611a419da023f9d0766886cd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/11515da1bdc7421395522dde6b85be9a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/05ad320028844df2b415b185fdfd8696] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=101.2 K 2023-04-28 04:58:20,570 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b86d91e4393d455995d6eb910321e8f3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1682657896277 2023-04-28 04:58:20,570 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 4da84585b32244a5af85281029e03066, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657896246 2023-04-28 04:58:20,571 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ebf9cedb71534e958d0de90e8b6288b9, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682657897401 2023-04-28 04:58:20,571 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 08e87db2611a419da023f9d0766886cd, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1682657896277 2023-04-28 04:58:20,573 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 11515da1bdc7421395522dde6b85be9a, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682657897401 2023-04-28 04:58:20,573 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 9c2d1681e1054c8c856f74110082b3a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682657898604 2023-04-28 04:58:20,573 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 05ad320028844df2b415b185fdfd8696, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682657898599 2023-04-28 04:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:20,582 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:20,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:20,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:20,582 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:20,582 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:20,586 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#B#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:20,588 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,588 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:20,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,589 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:20,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,589 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,590 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:20,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:20,599 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:20,599 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:20,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:20,599 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#A#compaction#459 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:20,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,599 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:20,599 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:20,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,599 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:20,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:20,600 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:20,606 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/4a7e7943b9b74b3fbaa0e3d982b695a9 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4a7e7943b9b74b3fbaa0e3d982b695a9 2023-04-28 04:58:20,610 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/B of a6bab1698a9b263a4df9a801810dbac0 into 4a7e7943b9b74b3fbaa0e3d982b695a9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:20,610 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:20,610 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/B, priority=12, startTime=1682657900567; duration=0sec 2023-04-28 04:58:20,610 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:20,610 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 04:58:20,611 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45790 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 04:58:20,611 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/C is initiating minor compaction (all files) 2023-04-28 04:58:20,611 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/C in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,611 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/03a4e99560544c299a883d7931d618e3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5302371a9bfc417f8ee513188d193e66, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/e356bc5a5f08421da7b443777b4c0c5e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5f7df0822f36431c97d7da32fb83e46c] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=44.7 K 2023-04-28 04:58:20,611 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 03a4e99560544c299a883d7931d618e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682657896246 2023-04-28 04:58:20,611 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5302371a9bfc417f8ee513188d193e66, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1682657896277 2023-04-28 04:58:20,612 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting e356bc5a5f08421da7b443777b4c0c5e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682657897401 2023-04-28 04:58:20,612 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 5f7df0822f36431c97d7da32fb83e46c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682657898604 2023-04-28 04:58:20,617 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#C#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:20,624 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/7f74c9d8acd54695a85f22b0ce7fae6b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/7f74c9d8acd54695a85f22b0ce7fae6b 2023-04-28 04:58:20,627 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/C of a6bab1698a9b263a4df9a801810dbac0 into 7f74c9d8acd54695a85f22b0ce7fae6b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:20,627 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:20,627 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/C, priority=12, startTime=1682657900568; duration=0sec 2023-04-28 04:58:20,628 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:20,700 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:20,702 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:20,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:20,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:20,703 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:20,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:20,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:20,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,703 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,714 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:20,714 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:20,715 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,715 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:20,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:20,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:20,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:20,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:20,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:20,716 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:20,720 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:20,720 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:20,720 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:20,720 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:20,720 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 04:58:20,721 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:20,721 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:20,721 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:20,721 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:20,721 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:20,721 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:20,736 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428902b33ea63834736a4917ca732139199 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428902b33ea63834736a4917ca732139199 2023-04-28 04:58:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:20,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:20,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657900855, max timestamp=1682657900863Num uniques 859; 2023-04-28 04:58:20,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682657900855, max timestamp=1682657900863Num uniques 859; 2023-04-28 04:58:20,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657900855, max timestamp=1682657900863Num uniques 859; 2023-04-28 04:58:20,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:20,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:20,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:20,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657960882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657960883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657960883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657960985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657960986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:20,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:20,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657960986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,140 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/38ba2c6b21d34b68868f51f8c6a89761 2023-04-28 04:58:21,158 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/c4cace55308f4722b21d2056c29ef14a 2023-04-28 04:58:21,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:21,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657961187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:21,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.104.168:47428 deadline: 1682657961188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:21,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657961189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:21,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657961491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:21,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657961492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657961493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,571 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/d1db41167f3d4160b97fe1ee6486c5a0 2023-04-28 04:58:21,574 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/38ba2c6b21d34b68868f51f8c6a89761 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/38ba2c6b21d34b68868f51f8c6a89761 2023-04-28 04:58:21,577 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/38ba2c6b21d34b68868f51f8c6a89761, entries=150, sequenceid=193, filesize=25.3 K 2023-04-28 04:58:21,577 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/c4cace55308f4722b21d2056c29ef14a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/c4cace55308f4722b21d2056c29ef14a 2023-04-28 04:58:21,580 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/c4cace55308f4722b21d2056c29ef14a, entries=150, sequenceid=193, filesize=11.8 K 2023-04-28 04:58:21,580 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/d1db41167f3d4160b97fe1ee6486c5a0 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/d1db41167f3d4160b97fe1ee6486c5a0 2023-04-28 04:58:21,583 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/d1db41167f3d4160b97fe1ee6486c5a0, entries=150, sequenceid=193, filesize=11.8 K 2023-04-28 04:58:21,583 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a6bab1698a9b263a4df9a801810dbac0 in 863ms, sequenceid=193, compaction requested=false 2023-04-28 04:58:21,583 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:21,583 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:21,583 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:21,583 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:21,583 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,583 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:21,584 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:21,648 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,648 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:21,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,649 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@310458eb[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:21,650 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:21,650 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,662 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,662 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:21,662 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:21,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,662 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:21,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:21,699 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:21,699 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:21,699 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,699 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:21,701 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:21,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:21,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:21,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:21,701 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:21,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:21,722 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,722 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:21,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:21,746 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:21,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:21,746 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:21,746 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:21,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,746 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:21,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:21,746 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:21,746 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:21,747 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,830 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/552b6c4fa8e64297b1be7fa472b68d8f as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/552b6c4fa8e64297b1be7fa472b68d8f 2023-04-28 04:58:21,833 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/A of a6bab1698a9b263a4df9a801810dbac0 into 552b6c4fa8e64297b1be7fa472b68d8f(size=25.7 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:58:21,833 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:21,833 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/A, priority=12, startTime=1682657900567; duration=1sec 2023-04-28 04:58:21,833 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:21,847 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:21,848 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:21,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:21,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:21,850 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:21,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:21,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:21,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:21,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,850 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:21,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:21,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:21,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:21,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:21,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:21,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:21,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:21,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:21,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:21,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:21,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:21,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:21,863 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:21,863 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:21,863 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:21,863 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 04:58:21,864 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:21,864 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:21,864 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:21,864 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:21,864 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:21,864 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:21,874 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428aac4042054c44bb6880996a15ade802f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aac4042054c44bb6880996a15ade802f 2023-04-28 04:58:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:21,995 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:22,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657962001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657962002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657962004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657962114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657962114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657962114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,278 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/791a2fc5b884455dacc202721d43dd23 2023-04-28 04:58:22,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657962317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657962317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657962318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657962620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657962621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:22,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657962621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,686 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/b2811535568e47e8920b4712a513d290 2023-04-28 04:58:22,693 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a99d215e5e044983b5f90fe6e62f4279 2023-04-28 04:58:22,696 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/791a2fc5b884455dacc202721d43dd23 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/791a2fc5b884455dacc202721d43dd23 2023-04-28 04:58:22,698 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/791a2fc5b884455dacc202721d43dd23, entries=150, sequenceid=231, filesize=25.3 K 2023-04-28 04:58:22,699 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/b2811535568e47e8920b4712a513d290 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b2811535568e47e8920b4712a513d290 2023-04-28 04:58:22,702 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b2811535568e47e8920b4712a513d290, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 04:58:22,702 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a99d215e5e044983b5f90fe6e62f4279 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a99d215e5e044983b5f90fe6e62f4279 2023-04-28 04:58:22,705 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a99d215e5e044983b5f90fe6e62f4279, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 04:58:22,705 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for a6bab1698a9b263a4df9a801810dbac0 in 842ms, sequenceid=231, compaction requested=true 2023-04-28 04:58:22,706 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:22,706 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:22,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:22,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:22,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:22,706 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:22,785 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:22,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,785 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4dadbe[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:22,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:22,786 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,798 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:22,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:22,799 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,799 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:22,799 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:22,809 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:22,809 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:22,809 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:22,809 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:22,811 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:22,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:22,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:22,811 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:22,811 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:22,819 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,819 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:22,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,819 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:22,820 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:22,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:22,830 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:22,830 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:22,830 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:22,830 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:22,830 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:22,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:22,831 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:22,930 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:22,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:22,932 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:22,933 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:22,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:22,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:22,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:22,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,933 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:22,943 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:22,943 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:22,945 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:22,945 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:22,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:22,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:22,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:22,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:22,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:22,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:22,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:22,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:22,946 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:22,946 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:22,946 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:22,955 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428871743296881476f9d91e995d2ca0c38 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428871743296881476f9d91e995d2ca0c38 2023-04-28 04:58:22,958 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/6bec7cf5e8e14dd1b5835a005af62d2b 2023-04-28 04:58:22,966 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/3ab3d54c985741b0825c3d72a1de881c 2023-04-28 04:58:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:23,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:23,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657903152, max timestamp=1682657903160Num uniques 859; 2023-04-28 04:58:23,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:23,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657903152, max timestamp=1682657903160Num uniques 859; 2023-04-28 04:58:23,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:23,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682657903152, max timestamp=1682657903160Num uniques 859; 2023-04-28 04:58:23,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:23,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657963179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657963180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657963183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657963284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657963285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657963286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,375 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/0e044f3c98204bf9ae743fe53bacbecc 2023-04-28 04:58:23,377 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/6bec7cf5e8e14dd1b5835a005af62d2b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6bec7cf5e8e14dd1b5835a005af62d2b 2023-04-28 04:58:23,381 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6bec7cf5e8e14dd1b5835a005af62d2b, entries=150, sequenceid=240, filesize=25.3 K 2023-04-28 04:58:23,382 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/3ab3d54c985741b0825c3d72a1de881c as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/3ab3d54c985741b0825c3d72a1de881c 2023-04-28 04:58:23,385 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/3ab3d54c985741b0825c3d72a1de881c, entries=150, sequenceid=240, filesize=11.8 K 2023-04-28 04:58:23,385 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/0e044f3c98204bf9ae743fe53bacbecc as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/0e044f3c98204bf9ae743fe53bacbecc 2023-04-28 04:58:23,387 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/0e044f3c98204bf9ae743fe53bacbecc, entries=150, sequenceid=240, filesize=11.8 K 2023-04-28 04:58:23,388 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for a6bab1698a9b263a4df9a801810dbac0 in 442ms, sequenceid=240, compaction requested=true 2023-04-28 04:58:23,388 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:23,388 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:23,388 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:23,388 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:23,388 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,388 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:23,388 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:23,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,438 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,438 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:23,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ceff4cc[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:23,439 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:23,439 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,451 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,451 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:23,451 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:23,451 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,451 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:23,451 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:23,461 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:23,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:23,461 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:23,463 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:23,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:23,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:23,464 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:23,464 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:23,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:23,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,473 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,483 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:23,483 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:23,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:23,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:23,483 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:23,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:23,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:23,483 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:23,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:23,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-28 04:58:23,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:23,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:58:23,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:23,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:58:23,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:23,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:58:23,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657963496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657963497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657963497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,583 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:23,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:23,585 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:23,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:23,587 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:23,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:23,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:23,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:23,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,587 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,598 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:23,598 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:23,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,599 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:23,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:23,600 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:23,600 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:23,600 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:23,600 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:23,600 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:23,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,610 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657963608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657963608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657963608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:23,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a92c532[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:23,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:23,611 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,619 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:23,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:23,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,620 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:23,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:23,630 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:23,630 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:23,630 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,630 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:23,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:23,632 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:23,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:23,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:23,632 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:23,632 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:23,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,668 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:23,669 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,670 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,690 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:23,690 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:23,690 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:23,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:23,691 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:23,690 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,691 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:23,691 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:23,691 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,791 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:23,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:23,793 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:23,794 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:23,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:23,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:23,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:23,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,794 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,806 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:23,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:23,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,806 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:23,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:23,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:23,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:23,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:23,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:23,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:23,807 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:23,807 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:23,808 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:23,808 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:23,808 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:23,808 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:23,808 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:23,808 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,808 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:23,808 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:23,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657963811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657963812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657963812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,819 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,819 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,820 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c21d8b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:23,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:23,821 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,830 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,830 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:23,830 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:23,830 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,830 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:23,830 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:23,840 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:23,840 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,841 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,841 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:23,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:23,843 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:23,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:23,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:23,843 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:23,843 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:23,851 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,851 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,851 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:23,852 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,853 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,862 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:23,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,862 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:23,862 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:23,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:23,862 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,862 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:23,862 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,900 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ed1008072e6d4d448f98c801ae9416d4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ed1008072e6d4d448f98c801ae9416d4 2023-04-28 04:58:23,903 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=278, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/5a3264cb1bc04116a172928d388f5c1e 2023-04-28 04:58:23,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/fe0153d478f84467a1496dbbaf8047a7 2023-04-28 04:58:23,962 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:23,965 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:23,987 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:23,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:23,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:23,987 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:23,988 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,988 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,998 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,998 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:23,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:23,999 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:23,999 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:23,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:24,000 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,000 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:24,000 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:24,000 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:24,000 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:24,009 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,009 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:24,009 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,009 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a616950[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:24,010 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:24,010 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,019 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,019 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:24,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:24,020 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:24,020 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,020 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:24,030 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:24,030 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:24,030 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,030 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:24,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:24,032 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:24,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:24,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:24,032 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:24,032 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:24,040 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,040 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:24,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,040 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:24,041 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:24,042 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,051 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:24,051 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,051 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:24,051 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:24,051 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:24,051 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:24,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:24,052 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:24,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657964115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657964115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657964116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,152 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:24,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:24,154 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:24,155 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:24,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:24,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:24,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:24,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,155 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,167 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:24,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:24,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,167 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:24,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:24,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:24,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:24,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:24,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:24,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:24,168 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:24,168 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,168 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:24,169 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:24,169 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:24,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:24,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:24,169 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:24,177 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,177 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:24,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,178 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690bad44[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:24,178 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:24,179 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,188 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,188 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:24,188 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:24,188 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:24,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,188 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:24,198 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:24,198 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:24,198 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,198 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:24,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:24,200 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:24,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:24,200 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:24,201 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:24,201 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:24,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:24,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,209 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:24,209 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,210 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:24,248 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:24,248 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:24,248 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:24,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:24,248 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:24,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,248 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:24,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:24,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:24,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:24,248 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,249 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,249 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:24,249 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:24,348 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:24,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/cf8695a8bedc47fc90f39c88bdd34343 2023-04-28 04:58:24,350 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:24,352 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:24,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:24,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:24,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:24,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,352 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/5a3264cb1bc04116a172928d388f5c1e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/5a3264cb1bc04116a172928d388f5c1e 2023-04-28 04:58:24,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/5a3264cb1bc04116a172928d388f5c1e, entries=200, sequenceid=278, filesize=32.3 K 2023-04-28 04:58:24,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/fe0153d478f84467a1496dbbaf8047a7 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/fe0153d478f84467a1496dbbaf8047a7 2023-04-28 04:58:24,359 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/fe0153d478f84467a1496dbbaf8047a7, entries=150, sequenceid=278, filesize=11.9 K 2023-04-28 04:58:24,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/cf8695a8bedc47fc90f39c88bdd34343 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/cf8695a8bedc47fc90f39c88bdd34343 2023-04-28 04:58:24,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/cf8695a8bedc47fc90f39c88bdd34343, entries=150, sequenceid=278, filesize=11.9 K 2023-04-28 04:58:24,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for a6bab1698a9b263a4df9a801810dbac0 in 874ms, sequenceid=278, compaction requested=true 2023-04-28 04:58:24,364 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:24,364 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:58:24,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:24,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:24,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 04:58:24,370 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:58:24,373 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137046 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:58:24,373 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/A is initiating minor compaction (all files) 2023-04-28 04:58:24,373 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/A in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,374 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60706 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:58:24,374 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/552b6c4fa8e64297b1be7fa472b68d8f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/38ba2c6b21d34b68868f51f8c6a89761, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/791a2fc5b884455dacc202721d43dd23, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6bec7cf5e8e14dd1b5835a005af62d2b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/5a3264cb1bc04116a172928d388f5c1e] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=133.8 K 2023-04-28 04:58:24,374 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/B is initiating minor compaction (all files) 2023-04-28 04:58:24,374 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/B in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,374 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4a7e7943b9b74b3fbaa0e3d982b695a9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/c4cace55308f4722b21d2056c29ef14a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b2811535568e47e8920b4712a513d290, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/3ab3d54c985741b0825c3d72a1de881c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/fe0153d478f84467a1496dbbaf8047a7] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=59.3 K 2023-04-28 04:58:24,374 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 552b6c4fa8e64297b1be7fa472b68d8f, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682657898604 2023-04-28 04:58:24,374 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4a7e7943b9b74b3fbaa0e3d982b695a9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682657898604 2023-04-28 04:58:24,374 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 38ba2c6b21d34b68868f51f8c6a89761, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682657899728 2023-04-28 04:58:24,374 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting c4cace55308f4722b21d2056c29ef14a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682657899728 2023-04-28 04:58:24,375 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 791a2fc5b884455dacc202721d43dd23, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682657900879 2023-04-28 04:58:24,375 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting b2811535568e47e8920b4712a513d290, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682657900879 2023-04-28 04:58:24,375 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6bec7cf5e8e14dd1b5835a005af62d2b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682657902000 2023-04-28 04:58:24,375 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 3ab3d54c985741b0825c3d72a1de881c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682657902000 2023-04-28 04:58:24,380 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 5a3264cb1bc04116a172928d388f5c1e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682657903177 2023-04-28 04:58:24,380 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting fe0153d478f84467a1496dbbaf8047a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682657903177 2023-04-28 04:58:24,394 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#B#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:24,398 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:24,398 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:24,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:24,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:24,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:24,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:24,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:24,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:24,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:24,404 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:24,404 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:24,404 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,404 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:24,404 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 04:58:24,405 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:24,405 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:24,405 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:24,405 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:24,405 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:24,405 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:24,409 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#A#compaction#474 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:24,414 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/089f0a5534f24115b3c1883a82efd7e4 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/089f0a5534f24115b3c1883a82efd7e4 2023-04-28 04:58:24,418 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/B of a6bab1698a9b263a4df9a801810dbac0 into 089f0a5534f24115b3c1883a82efd7e4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:24,418 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:24,418 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/B, priority=11, startTime=1682657904364; duration=0sec 2023-04-28 04:58:24,419 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:24,419 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 04:58:24,420 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60706 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 04:58:24,420 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/C is initiating minor compaction (all files) 2023-04-28 04:58:24,420 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/C in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:24,420 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/7f74c9d8acd54695a85f22b0ce7fae6b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/d1db41167f3d4160b97fe1ee6486c5a0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a99d215e5e044983b5f90fe6e62f4279, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/0e044f3c98204bf9ae743fe53bacbecc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/cf8695a8bedc47fc90f39c88bdd34343] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=59.3 K 2023-04-28 04:58:24,421 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7f74c9d8acd54695a85f22b0ce7fae6b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682657898604 2023-04-28 04:58:24,421 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting d1db41167f3d4160b97fe1ee6486c5a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682657899728 2023-04-28 04:58:24,422 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a99d215e5e044983b5f90fe6e62f4279, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682657900879 2023-04-28 04:58:24,424 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 0e044f3c98204bf9ae743fe53bacbecc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682657902000 2023-04-28 04:58:24,425 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting cf8695a8bedc47fc90f39c88bdd34343, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682657903177 2023-04-28 04:58:24,437 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#C#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:24,438 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042834ce01d826dc4c309d4e4f17d7b4e901 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042834ce01d826dc4c309d4e4f17d7b4e901 2023-04-28 04:58:24,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:24,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:24,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682657904634, max timestamp=1682657904641Num uniques 859; 2023-04-28 04:58:24,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682657904634, max timestamp=1682657904641Num uniques 859; 2023-04-28 04:58:24,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:24,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682657904634, max timestamp=1682657904641Num uniques 859; 2023-04-28 04:58:24,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:24,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 04:58:24,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657964663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682657904661, max timestamp=1682657904666Num uniques 1068; 2023-04-28 04:58:24,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682657904661, max timestamp=1682657904666Num uniques 1068; 2023-04-28 04:58:24,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-28 04:58:24,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-28 04:58:24,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682657904661, max timestamp=1682657904666Num uniques 1068; 2023-04-28 04:58:24,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-28 04:58:24,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657964705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682657904686, max timestamp=1682657904706Num uniques 1077; 2023-04-28 04:58:24,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682657904686, max timestamp=1682657904706Num uniques 1077; 2023-04-28 04:58:24,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682657904686, max timestamp=1682657904706Num uniques 1077; 2023-04-28 04:58:24,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-28 04:58:24,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-28 04:58:24,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-28 04:58:24,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682657904711, max timestamp=1682657904764Num uniques 1036; 2023-04-28 04:58:24,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-28 04:58:24,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682657904711, max timestamp=1682657904764Num uniques 1036; 2023-04-28 04:58:24,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682657904711, max timestamp=1682657904764Num uniques 1036; 2023-04-28 04:58:24,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-28 04:58:24,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-28 04:58:24,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657904785, max timestamp=1682657904795Num uniques 1045; 2023-04-28 04:58:24,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-28 04:58:24,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657904785, max timestamp=1682657904795Num uniques 1045; 2023-04-28 04:58:24,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682657904785, max timestamp=1682657904795Num uniques 1045; 2023-04-28 04:58:24,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-28 04:58:24,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-28 04:58:24,842 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/7b7d01c9bb3b45dd937b7e84381a56e5 2023-04-28 04:58:24,844 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/ee8a466914fb4e6ba32b792f8b479977 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/ee8a466914fb4e6ba32b792f8b479977 2023-04-28 04:58:24,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657964842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657964842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:24,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657904807, max timestamp=1682657904844Num uniques 1054; 2023-04-28 04:58:24,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657904807, max timestamp=1682657904844Num uniques 1054; 2023-04-28 04:58:24,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 04:58:24,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682657904807, max timestamp=1682657904844Num uniques 1054; 2023-04-28 04:58:24,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 04:58:24,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-28 04:58:24,851 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/C of a6bab1698a9b263a4df9a801810dbac0 into ee8a466914fb4e6ba32b792f8b479977(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:24,851 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:24,851 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/C, priority=11, startTime=1682657904364; duration=0sec 2023-04-28 04:58:24,851 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:24,852 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/7cc73c10da7c433ba27737e0350279c2 2023-04-28 04:58:24,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657904860, max timestamp=1682657904892Num uniques 1013; 2023-04-28 04:58:24,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-28 04:58:24,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657904860, max timestamp=1682657904892Num uniques 1013; 2023-04-28 04:58:24,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-28 04:58:24,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657904860, max timestamp=1682657904892Num uniques 1013; 2023-04-28 04:58:24,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-28 04:58:24,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682657904921, max timestamp=1682657904943Num uniques 1022; 2023-04-28 04:58:24,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-28 04:58:24,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682657904921, max timestamp=1682657904943Num uniques 1022; 2023-04-28 04:58:24,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-28 04:58:24,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682657904921, max timestamp=1682657904943Num uniques 1022; 2023-04-28 04:58:24,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-28 04:58:24,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657904951, max timestamp=1682657904965Num uniques 1031; 2023-04-28 04:58:24,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 04:58:24,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657904951, max timestamp=1682657904965Num uniques 1031; 2023-04-28 04:58:24,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 04:58:24,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657904951, max timestamp=1682657904965Num uniques 1031; 2023-04-28 04:58:24,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-28 04:58:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682657904982, max timestamp=1682657904985Num uniques 1090; 2023-04-28 04:58:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-28 04:58:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682657904982, max timestamp=1682657904985Num uniques 1090; 2023-04-28 04:58:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-28 04:58:24,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657904982, max timestamp=1682657904985Num uniques 1090; 2023-04-28 04:58:24,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=6870, suffix heap size=141328,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=150 2023-04-28 04:58:25,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682657904993, max timestamp=1682657905006Num uniques 1049; 2023-04-28 04:58:25,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682657904993, max timestamp=1682657905006Num uniques 1049; 2023-04-28 04:58:25,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-28 04:58:25,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-28 04:58:25,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682657904993, max timestamp=1682657905006Num uniques 1009; 2023-04-28 04:58:25,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9114, suffix heap size=132904,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=199 2023-04-28 04:58:25,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657965022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682657905018, max timestamp=1682657905024Num uniques 1058; 2023-04-28 04:58:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-28 04:58:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682657905018, max timestamp=1682657905024Num uniques 1058; 2023-04-28 04:58:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-28 04:58:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682657905018, max timestamp=1682657905024Num uniques 1058; 2023-04-28 04:58:25,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-28 04:58:25,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682657905052, max timestamp=1682657905062Num uniques 1067; 2023-04-28 04:58:25,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-28 04:58:25,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682657905052, max timestamp=1682657905062Num uniques 1067; 2023-04-28 04:58:25,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-28 04:58:25,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682657905052, max timestamp=1682657905062Num uniques 1067; 2023-04-28 04:58:25,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-28 04:58:25,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682657905073, max timestamp=1682657905092Num uniques 1026; 2023-04-28 04:58:25,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-28 04:58:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682657905073, max timestamp=1682657905092Num uniques 1026; 2023-04-28 04:58:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-28 04:58:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682657905073, max timestamp=1682657905092Num uniques 1026; 2023-04-28 04:58:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-28 04:58:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1682657905099, max timestamp=1682657905133Num uniques 1085; 2023-04-28 04:58:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1682657905099, max timestamp=1682657905133Num uniques 1085; 2023-04-28 04:58:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1682657905099, max timestamp=1682657905133Num uniques 1085; 2023-04-28 04:58:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-28 04:58:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-28 04:58:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-28 04:58:25,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657965152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657965155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657965155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657965256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,259 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/de2d03b12e7e4aeca505ff2a11b989a6 2023-04-28 04:58:25,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657965258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657965259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,263 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/7b7d01c9bb3b45dd937b7e84381a56e5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/7b7d01c9bb3b45dd937b7e84381a56e5 2023-04-28 04:58:25,265 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/7b7d01c9bb3b45dd937b7e84381a56e5, entries=100, sequenceid=286, filesize=18.6 K 2023-04-28 04:58:25,266 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/7cc73c10da7c433ba27737e0350279c2 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/7cc73c10da7c433ba27737e0350279c2 2023-04-28 04:58:25,269 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/7cc73c10da7c433ba27737e0350279c2, entries=100, sequenceid=286, filesize=9.5 K 2023-04-28 04:58:25,269 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/de2d03b12e7e4aeca505ff2a11b989a6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/de2d03b12e7e4aeca505ff2a11b989a6 2023-04-28 04:58:25,271 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/de2d03b12e7e4aeca505ff2a11b989a6, entries=100, sequenceid=286, filesize=9.5 K 2023-04-28 04:58:25,272 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for a6bab1698a9b263a4df9a801810dbac0 in 868ms, sequenceid=286, compaction requested=false 2023-04-28 04:58:25,272 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:25,272 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:25,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:25,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:25,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:25,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,346 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:25,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25fc5e02[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:25,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:25,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,356 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:25,356 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:25,356 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:25,356 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,356 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:25,367 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:25,367 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,367 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,367 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:25,369 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:25,369 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:25,369 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:25,369 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:25,369 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:25,377 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,377 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,378 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,378 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,379 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,388 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:25,388 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:25,388 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,388 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,388 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:25,388 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:25,464 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=154.31 KB heapSize=433.64 KB 2023-04-28 04:58:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:58:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:58:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:25,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 04:58:25,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657965466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657965467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657965468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,489 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:25,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:25,490 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:25,491 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:25,491 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:25,491 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:25,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:25,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,492 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,504 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,504 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:25,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:25,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:25,505 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,505 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:25,505 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:25,505 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:25,505 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:25,514 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,514 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,514 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,514 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:25,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8b2173[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:25,515 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:25,515 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:25,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:25,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:25,525 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:25,535 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:25,535 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,535 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,535 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:25,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:25,537 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:25,537 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:25,537 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:25,537 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:25,537 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:25,546 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,546 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,546 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,546 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,547 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,556 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:25,557 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:25,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:25,556 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,557 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:25,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,556 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,557 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:25,557 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657965569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657965571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657965571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,640 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/6884d6ced2d7466294f0bb29b8a2ee19 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6884d6ced2d7466294f0bb29b8a2ee19 2023-04-28 04:58:25,643 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/A of a6bab1698a9b263a4df9a801810dbac0 into 6884d6ced2d7466294f0bb29b8a2ee19(size=26.0 K), total size for store is 44.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:58:25,643 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:25,643 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/A, priority=11, startTime=1682657904364; duration=1sec 2023-04-28 04:58:25,643 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:25,657 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:25,659 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:25,659 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:25,659 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:25,659 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:25,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:25,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,660 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,672 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,672 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,672 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,672 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:25,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:25,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:25,673 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:25,673 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:25,673 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:25,674 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,674 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:25,674 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:25,674 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:25,674 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:25,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:25,688 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:25,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d61fc5b[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:25,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:25,689 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,701 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:25,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:25,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,701 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:25,701 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:25,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:25,711 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,711 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:25,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:25,713 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:25,713 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:25,713 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:25,713 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:25,713 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:25,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,725 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,725 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,726 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,766 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,766 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,766 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,766 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657965772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657965773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:25,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657965774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,866 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:25,868 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:25,869 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:25,869 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:25,869 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:25,869 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:25,869 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,869 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,875 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042895e44f74dd62484eb6f1cc38863bfef6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042895e44f74dd62484eb6f1cc38863bfef6 2023-04-28 04:58:25,879 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=567, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/f0401ebed23442feb3e65f4fe28bce7b 2023-04-28 04:58:25,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,890 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,890 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:25,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:25,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:25,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:25,891 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:25,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:25,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:25,892 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:25,892 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:25,904 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,904 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,904 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,904 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b4173e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:25,905 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:25,905 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,914 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,914 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:25,914 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:25,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,914 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:25,914 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:25,925 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,925 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:25,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:25,927 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:25,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:25,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:25,927 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:25,927 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:25,935 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,935 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:25,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,935 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:25,936 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:25,946 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:25,946 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:25,946 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:25,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:25,946 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:25,946 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:25,946 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,046 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:26,048 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:26,049 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:26,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:26,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:26,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,049 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,062 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,062 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,062 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,062 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:26,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:26,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:26,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:26,064 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:26,065 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,065 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:26,065 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:26,065 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:26,065 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:26,072 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,072 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:26,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eff4444[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:26,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:26,074 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:26,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657966074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:26,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657966076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:26,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657966077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,082 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,083 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:26,083 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:26,083 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:26,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,083 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:26,093 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:26,093 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,093 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:26,093 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:26,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:26,095 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:26,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:26,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:26,095 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:26,095 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:26,103 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,104 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,104 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,104 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,105 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,114 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,114 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:26,114 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:26,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:26,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,115 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:26,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,115 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:26,115 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,215 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:26,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:26,217 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:26,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:26,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:26,218 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:26,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:26,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:26,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,218 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,230 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,230 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:26,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:26,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:26,231 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:26,231 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,231 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:26,232 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,232 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:26,232 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:26,232 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:26,232 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:26,240 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,240 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,241 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,241 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:26,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bda6451[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:26,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:26,242 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,251 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,251 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:26,251 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,251 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:26,251 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:26,251 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:26,261 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,261 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:26,262 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,262 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:26,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:26,263 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:26,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:26,263 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:26,264 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:26,264 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:26,272 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,272 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,272 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,273 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,273 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:26,283 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:26,283 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,283 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,283 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:26,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,283 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,283 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/58129b7d7aca49699950922c1aaec3b5 2023-04-28 04:58:26,383 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:26,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:26,385 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:26,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:26,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:26,386 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:26,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:26,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:26,386 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,387 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,398 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,398 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,399 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:26,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:26,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:26,400 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,400 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:26,400 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:26,400 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:26,400 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:26,440 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,440 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,440 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,440 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596bce55[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:26,441 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:26,442 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,461 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,462 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:26,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:26,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:26,462 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:26,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:26,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,472 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:26,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:26,474 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:26,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:26,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:26,474 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:26,474 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:26,482 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,483 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,483 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:26,493 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:26,493 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:26,493 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,494 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:26,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,494 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,494 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:26,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657966576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:26,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657966582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:26,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657966582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,594 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:26,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:26,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:26,597 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:26,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:26,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:26,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,597 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,609 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,609 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:26,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:26,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:26,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:26,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:26,610 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:26,611 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,611 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:26,611 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:26,611 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:26,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:26,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:26,611 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:26,619 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,619 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,620 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:26,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704f2fa1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:26,621 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:26,621 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,630 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,630 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:26,630 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:26,630 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,630 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:26,630 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:26,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:26,640 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,641 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:26,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:26,642 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:26,642 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:26,642 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:26,642 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:26,642 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:26,651 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,651 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:26,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,651 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:26,651 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,652 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,661 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,661 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,661 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,662 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:26,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:26,662 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,662 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:26,662 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,661 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:26,662 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,662 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,662 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:26,662 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,662 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:26,662 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:26,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a8bda8afb53b4b2f907e279ccb613b89 2023-04-28 04:58:26,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/f0401ebed23442feb3e65f4fe28bce7b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/f0401ebed23442feb3e65f4fe28bce7b 2023-04-28 04:58:26,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/f0401ebed23442feb3e65f4fe28bce7b, entries=200, sequenceid=567, filesize=32.3 K 2023-04-28 04:58:26,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/58129b7d7aca49699950922c1aaec3b5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/58129b7d7aca49699950922c1aaec3b5 2023-04-28 04:58:26,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/58129b7d7aca49699950922c1aaec3b5, entries=150, sequenceid=567, filesize=11.9 K 2023-04-28 04:58:26,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a8bda8afb53b4b2f907e279ccb613b89 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a8bda8afb53b4b2f907e279ccb613b89 2023-04-28 04:58:26,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a8bda8afb53b4b2f907e279ccb613b89, entries=150, sequenceid=567, filesize=11.9 K 2023-04-28 04:58:26,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~453.52 KB/464400, currentSize=20.13 KB/20610 for a6bab1698a9b263a4df9a801810dbac0 in 1249ms, sequenceid=567, compaction requested=true 2023-04-28 04:58:26,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:26,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:26,714 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:26,714 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:26,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:26,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:26,714 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78676 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:26,715 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:26,715 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/A is initiating minor compaction (all files) 2023-04-28 04:58:26,715 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/B is initiating minor compaction (all files) 2023-04-28 04:58:26,715 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/A in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,715 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/B in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,715 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6884d6ced2d7466294f0bb29b8a2ee19, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/7b7d01c9bb3b45dd937b7e84381a56e5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/f0401ebed23442feb3e65f4fe28bce7b] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=76.8 K 2023-04-28 04:58:26,715 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/089f0a5534f24115b3c1883a82efd7e4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/7cc73c10da7c433ba27737e0350279c2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/58129b7d7aca49699950922c1aaec3b5] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=33.8 K 2023-04-28 04:58:26,715 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 6884d6ced2d7466294f0bb29b8a2ee19, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682657903177 2023-04-28 04:58:26,715 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 089f0a5534f24115b3c1883a82efd7e4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682657903177 2023-04-28 04:58:26,716 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 7b7d01c9bb3b45dd937b7e84381a56e5, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1682657903495 2023-04-28 04:58:26,716 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 7cc73c10da7c433ba27737e0350279c2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1682657903495 2023-04-28 04:58:26,716 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting f0401ebed23442feb3e65f4fe28bce7b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1682657905153 2023-04-28 04:58:26,716 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 58129b7d7aca49699950922c1aaec3b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1682657905153 2023-04-28 04:58:26,729 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#B#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:26,737 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#A#compaction#482 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:26,762 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:26,764 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:26,765 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:26,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:26,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:26,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,765 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,777 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:26,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:26,777 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,778 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:26,778 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:26,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:26,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:26,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:26,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:26,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:26,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:26,779 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:26,780 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:26,780 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:26,780 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:26,780 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:26,780 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 04:58:26,780 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:26,780 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:26,780 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:26,780 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:26,781 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:26,781 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:26,800 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304286f7bfa3ec5a7443db5200e13d3e3cfda to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286f7bfa3ec5a7443db5200e13d3e3cfda 2023-04-28 04:58:26,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:26,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:26,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657906906, max timestamp=1682657906923Num uniques 859; 2023-04-28 04:58:26,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:58:26,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657906906, max timestamp=1682657906923Num uniques 859; 2023-04-28 04:58:26,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:58:26,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657906906, max timestamp=1682657906923Num uniques 859; 2023-04-28 04:58:26,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:58:26,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682657906953, max timestamp=1682657906963Num uniques 1018; 2023-04-28 04:58:26,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 04:58:26,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682657906953, max timestamp=1682657906963Num uniques 1018; 2023-04-28 04:58:26,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 04:58:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682657906953, max timestamp=1682657906963Num uniques 1018; 2023-04-28 04:58:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 04:58:26,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657906975, max timestamp=1682657906989Num uniques 1077; 2023-04-28 04:58:26,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 04:58:26,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657906975, max timestamp=1682657906989Num uniques 1077; 2023-04-28 04:58:26,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682657906975, max timestamp=1682657906989Num uniques 1077; 2023-04-28 04:58:26,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 04:58:26,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 04:58:27,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657907007, max timestamp=1682657907015Num uniques 1086; 2023-04-28 04:58:27,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 04:58:27,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657907007, max timestamp=1682657907015Num uniques 1086; 2023-04-28 04:58:27,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 04:58:27,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682657907007, max timestamp=1682657907015Num uniques 1086; 2023-04-28 04:58:27,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-28 04:58:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682657907032, max timestamp=1682657907042Num uniques 1095; 2023-04-28 04:58:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 04:58:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682657907032, max timestamp=1682657907042Num uniques 1095; 2023-04-28 04:58:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682657907032, max timestamp=1682657907042Num uniques 1095; 2023-04-28 04:58:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 04:58:27,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-28 04:58:27,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657907055, max timestamp=1682657907073Num uniques 1104; 2023-04-28 04:58:27,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 04:58:27,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657907055, max timestamp=1682657907073Num uniques 1104; 2023-04-28 04:58:27,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682657907055, max timestamp=1682657907073Num uniques 1104; 2023-04-28 04:58:27,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 04:58:27,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-28 04:58:27,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657907093, max timestamp=1682657907105Num uniques 1063; 2023-04-28 04:58:27,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657907093, max timestamp=1682657907105Num uniques 1063; 2023-04-28 04:58:27,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-28 04:58:27,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682657907093, max timestamp=1682657907105Num uniques 1063; 2023-04-28 04:58:27,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-28 04:58:27,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-28 04:58:27,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682657907125, max timestamp=1682657907135Num uniques 1022; 2023-04-28 04:58:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682657907125, max timestamp=1682657907135Num uniques 1022; 2023-04-28 04:58:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682657907125, max timestamp=1682657907135Num uniques 1022; 2023-04-28 04:58:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-28 04:58:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-28 04:58:27,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-28 04:58:27,146 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/110c9f3ff9634237a626cb1c1d5694a6 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/110c9f3ff9634237a626cb1c1d5694a6 2023-04-28 04:58:27,150 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/B of a6bab1698a9b263a4df9a801810dbac0 into 110c9f3ff9634237a626cb1c1d5694a6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:27,150 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:27,150 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/B, priority=13, startTime=1682657906714; duration=0sec 2023-04-28 04:58:27,151 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:27,151 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:27,152 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:27,152 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/C is initiating minor compaction (all files) 2023-04-28 04:58:27,152 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/C in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:27,152 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/ee8a466914fb4e6ba32b792f8b479977, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/de2d03b12e7e4aeca505ff2a11b989a6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a8bda8afb53b4b2f907e279ccb613b89] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=33.8 K 2023-04-28 04:58:27,152 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting ee8a466914fb4e6ba32b792f8b479977, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682657903177 2023-04-28 04:58:27,153 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting de2d03b12e7e4aeca505ff2a11b989a6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1682657903495 2023-04-28 04:58:27,153 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting a8bda8afb53b4b2f907e279ccb613b89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1682657905153 2023-04-28 04:58:27,168 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#C#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657907152, max timestamp=1682657907172Num uniques 1081; 2023-04-28 04:58:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 04:58:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657907152, max timestamp=1682657907172Num uniques 1081; 2023-04-28 04:58:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 04:58:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682657907152, max timestamp=1682657907172Num uniques 1081; 2023-04-28 04:58:27,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-28 04:58:27,176 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/be821825be594044930b785ddeef0ad5 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/be821825be594044930b785ddeef0ad5 2023-04-28 04:58:27,179 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/C of a6bab1698a9b263a4df9a801810dbac0 into be821825be594044930b785ddeef0ad5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:27,180 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:27,180 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/C, priority=13, startTime=1682657906714; duration=0sec 2023-04-28 04:58:27,180 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:27,206 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=574, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/19647301c9da465ead4d80c9ef641f06 2023-04-28 04:58:27,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682657907198, max timestamp=1682657907221Num uniques 1090; 2023-04-28 04:58:27,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-28 04:58:27,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1682657907198, max timestamp=1682657907221Num uniques 1090; 2023-04-28 04:58:27,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1682657907198, max timestamp=1682657907221Num uniques 1090; 2023-04-28 04:58:27,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=8690, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-28 04:58:27,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=8690, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-28 04:58:27,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682657907235, max timestamp=1682657907248Num uniques 1049; 2023-04-28 04:58:27,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-28 04:58:27,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682657907235, max timestamp=1682657907248Num uniques 1049; 2023-04-28 04:58:27,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-28 04:58:27,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682657907235, max timestamp=1682657907248Num uniques 1049; 2023-04-28 04:58:27,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-28 04:58:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682657907261, max timestamp=1682657907278Num uniques 1058; 2023-04-28 04:58:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-28 04:58:27,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682657907261, max timestamp=1682657907278Num uniques 1058; 2023-04-28 04:58:27,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-28 04:58:27,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682657907261, max timestamp=1682657907278Num uniques 1058; 2023-04-28 04:58:27,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-28 04:58:27,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682657907300, max timestamp=1682657907309Num uniques 1017; 2023-04-28 04:58:27,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-28 04:58:27,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682657907300, max timestamp=1682657907309Num uniques 1017; 2023-04-28 04:58:27,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682657907300, max timestamp=1682657907309Num uniques 1017; 2023-04-28 04:58:27,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-28 04:58:27,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-28 04:58:27,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682657907323, max timestamp=1682657907332Num uniques 1076; 2023-04-28 04:58:27,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682657907323, max timestamp=1682657907332Num uniques 1076; 2023-04-28 04:58:27,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682657907323, max timestamp=1682657907332Num uniques 1076; 2023-04-28 04:58:27,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-28 04:58:27,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-28 04:58:27,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-28 04:58:27,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682657907349, max timestamp=1682657907356Num uniques 1085; 2023-04-28 04:58:27,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-28 04:58:27,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682657907349, max timestamp=1682657907356Num uniques 1085; 2023-04-28 04:58:27,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682657907349, max timestamp=1682657907356Num uniques 1085; 2023-04-28 04:58:27,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-28 04:58:27,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-28 04:58:27,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682657907377, max timestamp=1682657907386Num uniques 1094; 2023-04-28 04:58:27,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-28 04:58:27,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682657907377, max timestamp=1682657907386Num uniques 1094; 2023-04-28 04:58:27,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682657907377, max timestamp=1682657907386Num uniques 1094; 2023-04-28 04:58:27,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-28 04:58:27,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-28 04:58:27,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682657907409, max timestamp=1682657907416Num uniques 1053; 2023-04-28 04:58:27,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-28 04:58:27,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682657907409, max timestamp=1682657907416Num uniques 1053; 2023-04-28 04:58:27,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682657907409, max timestamp=1682657907416Num uniques 1053; 2023-04-28 04:58:27,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-28 04:58:27,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-28 04:58:27,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682657907434, max timestamp=1682657907453Num uniques 1062; 2023-04-28 04:58:27,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-28 04:58:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682657907434, max timestamp=1682657907453Num uniques 1062; 2023-04-28 04:58:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682657907434, max timestamp=1682657907453Num uniques 1062; 2023-04-28 04:58:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-28 04:58:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-28 04:58:27,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682657907473, max timestamp=1682657907481Num uniques 1071; 2023-04-28 04:58:27,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-28 04:58:27,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682657907473, max timestamp=1682657907481Num uniques 1071; 2023-04-28 04:58:27,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-28 04:58:27,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682657907473, max timestamp=1682657907481Num uniques 1071; 2023-04-28 04:58:27,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-28 04:58:27,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1682657907504, max timestamp=1682657907510Num uniques 1080; 2023-04-28 04:58:27,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1682657907504, max timestamp=1682657907510Num uniques 1080; 2023-04-28 04:58:27,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-28 04:58:27,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1682657907504, max timestamp=1682657907510Num uniques 1080; 2023-04-28 04:58:27,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-28 04:58:27,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-28 04:58:27,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682657907533, max timestamp=1682657907538Num uniques 1089; 2023-04-28 04:58:27,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-28 04:58:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682657907533, max timestamp=1682657907538Num uniques 1089; 2023-04-28 04:58:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-28 04:58:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682657907533, max timestamp=1682657907538Num uniques 1089; 2023-04-28 04:58:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-28 04:58:27,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682657907553, max timestamp=1682657907560Num uniques 1048; 2023-04-28 04:58:27,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682657907553, max timestamp=1682657907560Num uniques 1048; 2023-04-28 04:58:27,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-28 04:58:27,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-28 04:58:27,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682657907553, max timestamp=1682657907560Num uniques 1048; 2023-04-28 04:58:27,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-28 04:58:27,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:27,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47440 deadline: 1682657967588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:27,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9462, totalHeapSize=21768, min timestamp=1682657907579, max timestamp=1682657907594Num uniques 1107; 2023-04-28 04:58:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=50682, new segment data size=9462, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-28 04:58:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8098, totalHeapSize=18648, min timestamp=1682657907579, max timestamp=1682657907594Num uniques 1107; 2023-04-28 04:58:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9462, totalHeapSize=21768, min timestamp=1682657907579, max timestamp=1682657907594Num uniques 1107; 2023-04-28 04:58:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=50688, new segment data size=8098, suffix heap size=143096,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=177 2023-04-28 04:58:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=50682, new segment data size=9462, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-28 04:58:27,615 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/317315fe310647d19e0fe3aaea1d2480 2023-04-28 04:58:27,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1682657907634, max timestamp=1682657907640Num uniques 1066; 2023-04-28 04:58:27,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-28 04:58:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1682657907634, max timestamp=1682657907640Num uniques 1066; 2023-04-28 04:58:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-28 04:58:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1682657907634, max timestamp=1682657907640Num uniques 1036; 2023-04-28 04:58:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=12176, suffix heap size=135712,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=266 2023-04-28 04:58:27,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1682657907652, max timestamp=1682657907659Num uniques 1125; 2023-04-28 04:58:27,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-28 04:58:27,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1682657907652, max timestamp=1682657907659Num uniques 1125; 2023-04-28 04:58:27,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-28 04:58:27,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1682657907652, max timestamp=1682657907659Num uniques 1125; 2023-04-28 04:58:27,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-28 04:58:27,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1682657907673, max timestamp=1682657907677Num uniques 1084; 2023-04-28 04:58:27,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-28 04:58:27,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1682657907673, max timestamp=1682657907677Num uniques 1084; 2023-04-28 04:58:27,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-28 04:58:27,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1682657907673, max timestamp=1682657907677Num uniques 1084; 2023-04-28 04:58:27,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-28 04:58:27,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682657907687, max timestamp=1682657907691Num uniques 1093; 2023-04-28 04:58:27,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=343, cellSize=15708, totalHeapSize=35912, min timestamp=1682657907685, max timestamp=1682657907691Num uniques 1093; 2023-04-28 04:58:27,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-28 04:58:27,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682657907687, max timestamp=1682657907691Num uniques 1093; 2023-04-28 04:58:27,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=15708, suffix heap size=141640,new segment heap size=35912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=343 2023-04-28 04:58:27,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-28 04:58:27,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:27,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657967709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:27,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1682657907703, max timestamp=1682657907711Num uniques 1202; 2023-04-28 04:58:27,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=55050, new segment data size=9250, suffix heap size=152976,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1202, new segment cells count=202 2023-04-28 04:58:27,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1682657907703, max timestamp=1682657907711Num uniques 1102; 2023-04-28 04:58:27,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1682657907703, max timestamp=1682657907711Num uniques 1102; 2023-04-28 04:58:27,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-28 04:58:27,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-28 04:58:27,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1682657907724, max timestamp=1682657907734Num uniques 1061; 2023-04-28 04:58:27,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-28 04:58:27,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1682657907724, max timestamp=1682657907734Num uniques 1061; 2023-04-28 04:58:27,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-28 04:58:27,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1682657907724, max timestamp=1682657907734Num uniques 1061; 2023-04-28 04:58:27,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-28 04:58:27,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1682657907769, max timestamp=1682657907775Num uniques 1020; 2023-04-28 04:58:27,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-28 04:58:27,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1682657907769, max timestamp=1682657907775Num uniques 1020; 2023-04-28 04:58:27,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-28 04:58:27,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1682657907769, max timestamp=1682657907775Num uniques 1020; 2023-04-28 04:58:27,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-28 04:58:27,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682657907789, max timestamp=1682657907798Num uniques 1079; 2023-04-28 04:58:27,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682657907789, max timestamp=1682657907798Num uniques 1079; 2023-04-28 04:58:27,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=10484, suffix heap size=140184,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=229 2023-04-28 04:58:27,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682657907789, max timestamp=1682657907798Num uniques 1079; 2023-04-28 04:58:27,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=10484, suffix heap size=140184,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=229 2023-04-28 04:58:27,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=10484, suffix heap size=140184,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=229 2023-04-28 04:58:27,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:27,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47464 deadline: 1682657967821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:27,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8589, totalHeapSize=19792, min timestamp=1682657907814, max timestamp=1682657907827Num uniques 1088; 2023-04-28 04:58:27,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=49809, new segment data size=8589, suffix heap size=141120,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=188 2023-04-28 04:58:27,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8607, totalHeapSize=19792, min timestamp=1682657907814, max timestamp=1682657907824Num uniques 1088; 2023-04-28 04:58:27,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=49827, new segment data size=8607, suffix heap size=141120,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=188 2023-04-28 04:58:27,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1682657907814, max timestamp=1682657907820Num uniques 1088; 2023-04-28 04:58:27,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=8608, suffix heap size=141120,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=188 2023-04-28 04:58:27,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1682657907865, max timestamp=1682657907871Num uniques 1047; 2023-04-28 04:58:27,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=11312, suffix heap size=136856,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=247 2023-04-28 04:58:27,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1682657907865, max timestamp=1682657907871Num uniques 1047; 2023-04-28 04:58:27,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=11312, suffix heap size=136856,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=247 2023-04-28 04:58:27,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1682657907865, max timestamp=1682657907871Num uniques 1047; 2023-04-28 04:58:27,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=11312, suffix heap size=136856,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=247 2023-04-28 04:58:27,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1682657907889, max timestamp=1682657907895Num uniques 1106; 2023-04-28 04:58:27,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-28 04:58:27,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1682657907889, max timestamp=1682657907895Num uniques 1106; 2023-04-28 04:58:27,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-28 04:58:27,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1682657907889, max timestamp=1682657907895Num uniques 1106; 2023-04-28 04:58:27,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-28 04:58:27,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1682657907905, max timestamp=1682657907921Num uniques 1015; 2023-04-28 04:58:27,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1682657907905, max timestamp=1682657907921Num uniques 1015; 2023-04-28 04:58:27,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=12130, suffix heap size=133528,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=265 2023-04-28 04:58:27,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=12130, suffix heap size=133528,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=265 2023-04-28 04:58:27,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1682657907905, max timestamp=1682657907921Num uniques 1015; 2023-04-28 04:58:27,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-28 04:58:27,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1682657907947, max timestamp=1682657907953Num uniques 1124; 2023-04-28 04:58:27,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=51474, new segment data size=10254, suffix heap size=144864,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=224 2023-04-28 04:58:27,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1682657907947, max timestamp=1682657907953Num uniques 1124; 2023-04-28 04:58:27,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=51474, new segment data size=10254, suffix heap size=144864,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=224 2023-04-28 04:58:27,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682657907947, max timestamp=1682657907953Num uniques 1024; 2023-04-28 04:58:27,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=6870, suffix heap size=134464,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=150 2023-04-28 04:58:27,965 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/a434e386affb4811b69ed3d2fe48ff38 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a434e386affb4811b69ed3d2fe48ff38 2023-04-28 04:58:27,968 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/A of a6bab1698a9b263a4df9a801810dbac0 into a434e386affb4811b69ed3d2fe48ff38(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:58:27,969 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:27,969 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/A, priority=13, startTime=1682657906714; duration=1sec 2023-04-28 04:58:27,969 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:27,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1682657907972, max timestamp=1682657907977Num uniques 1083; 2023-04-28 04:58:27,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=10668, suffix heap size=140600,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=233 2023-04-28 04:58:27,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1682657907972, max timestamp=1682657907977Num uniques 1083; 2023-04-28 04:58:27,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1682657907972, max timestamp=1682657907977Num uniques 1009; 2023-04-28 04:58:27,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=10668, suffix heap size=140600,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=233 2023-04-28 04:58:27,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10668, suffix heap size=132904,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=233 2023-04-28 04:58:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1682657907989, max timestamp=1682657907993Num uniques 1092; 2023-04-28 04:58:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=11082, suffix heap size=141536,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=242 2023-04-28 04:58:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1682657907989, max timestamp=1682657907993Num uniques 1092; 2023-04-28 04:58:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1682657907989, max timestamp=1682657907993Num uniques 1092; 2023-04-28 04:58:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=11082, suffix heap size=141536,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=242 2023-04-28 04:58:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-3] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=11082, suffix heap size=141536,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=242 2023-04-28 04:58:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9205, totalHeapSize=21144, min timestamp=1682657908009, max timestamp=1682657908014Num uniques 1101; 2023-04-28 04:58:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9205, totalHeapSize=21144, min timestamp=1682657908009, max timestamp=1682657908014Num uniques 1101; 2023-04-28 04:58:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-6] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=9205, suffix heap size=142472,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=201 2023-04-28 04:58:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-7] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=9205, suffix heap size=142472,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=201 2023-04-28 04:58:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9205, totalHeapSize=21144, min timestamp=1682657908009, max timestamp=1682657908014Num uniques 1101; 2023-04-28 04:58:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-4] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=9205, suffix heap size=142472,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=201 2023-04-28 04:58:28,022 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/752e20441a9a499daecb5d5dff730106 2023-04-28 04:58:28,026 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/19647301c9da465ead4d80c9ef641f06 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/19647301c9da465ead4d80c9ef641f06 2023-04-28 04:58:28,028 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/19647301c9da465ead4d80c9ef641f06, entries=150, sequenceid=574, filesize=25.4 K 2023-04-28 04:58:28,029 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/317315fe310647d19e0fe3aaea1d2480 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/317315fe310647d19e0fe3aaea1d2480 2023-04-28 04:58:28,031 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/317315fe310647d19e0fe3aaea1d2480, entries=150, sequenceid=574, filesize=11.9 K 2023-04-28 04:58:28,032 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/752e20441a9a499daecb5d5dff730106 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/752e20441a9a499daecb5d5dff730106 2023-04-28 04:58:28,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1682657908025, max timestamp=1682657908031Num uniques 1060; 2023-04-28 04:58:28,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-9] regionserver.CompactionPipeline(163): Suffix data size=48540, new segment data size=11900, suffix heap size=138208,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=260 2023-04-28 04:58:28,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1682657908025, max timestamp=1682657908031Num uniques 1060; 2023-04-28 04:58:28,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-1] regionserver.CompactionPipeline(163): Suffix data size=48540, new segment data size=11900, suffix heap size=138208,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=260 2023-04-28 04:58:28,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1682657908025, max timestamp=1682657908031Num uniques 1060; 2023-04-28 04:58:28,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-8] regionserver.CompactionPipeline(163): Suffix data size=48540, new segment data size=11900, suffix heap size=138208,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=260 2023-04-28 04:58:28,035 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/752e20441a9a499daecb5d5dff730106, entries=150, sequenceid=574, filesize=11.9 K 2023-04-28 04:58:28,036 INFO [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=53.67 KB/54960 for a6bab1698a9b263a4df9a801810dbac0 in 1256ms, sequenceid=574, compaction requested=false 2023-04-28 04:58:28,036 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:28,036 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,036 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:28,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:28,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=60.38 KB heapSize=156.38 KB 2023-04-28 04:58:28,036 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:28,036 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:28,036 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:28,036 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:28,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:28,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:28,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:28,047 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042805cfed0e61834009aa2d08475f4e956f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042805cfed0e61834009aa2d08475f4e956f 2023-04-28 04:58:28,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47380 deadline: 1682657968054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657908051, max timestamp=1682657908056Num uniques 859; 2023-04-28 04:58:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657908051, max timestamp=1682657908056Num uniques 859; 2023-04-28 04:58:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:58:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682657908051, max timestamp=1682657908056Num uniques 859; 2023-04-28 04:58:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:58:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase12:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 04:58:28,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657968079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47464 deadline: 1682657968080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657968081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,100 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,100 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,101 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:28,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44a69888[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:28,102 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:28,102 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,114 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:28,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:28,114 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:28,114 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,114 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:28,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:28,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,125 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:28,125 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:28,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:28,127 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:28,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:28,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:28,127 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:28,127 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:28,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,135 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,136 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,136 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,146 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:28,146 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,146 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,146 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,146 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,146 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47380 deadline: 1682657968158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657968182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47464 deadline: 1682657968183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657968184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,246 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:28,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:28,248 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:28,249 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:28,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:28,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:28,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:28,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,249 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,261 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,261 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,262 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:28,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:28,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:28,263 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,263 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:28,263 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:28,263 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:28,263 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:28,272 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,272 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce684a1[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:28,273 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:28,274 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,313 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,313 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,313 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:28,313 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:28,313 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:28,314 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:28,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:28,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,335 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,335 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:28,337 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:28,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:28,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:28,337 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:28,337 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:28,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,346 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,346 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,347 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,356 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,356 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:28,356 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:28,356 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:28,357 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,357 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,357 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,357 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:28,357 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:28,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47380 deadline: 1682657968361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37663] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657968385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47464 deadline: 1682657968386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657968387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,451 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1271, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/a1f0d02b0a204b15aa1bbd6ddf95fe27 2023-04-28 04:58:28,457 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:28,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:28,458 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:28,459 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:28,459 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:28,459 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:28,459 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:28,460 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,460 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,472 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,472 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:28,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:28,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:28,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:28,473 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:28,474 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,474 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:28,474 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:28,474 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:28,474 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:28,482 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,482 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,483 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,483 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:28,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf8e40e[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:28,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:28,484 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,493 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,493 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:28,493 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:28,493 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,493 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:28,493 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:28,503 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:28,503 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,503 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,504 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:28,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:28,505 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:28,506 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:28,506 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:28,506 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:28,506 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:28,514 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,514 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,514 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,514 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,515 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,525 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,525 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:28,525 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,525 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:28,525 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:28,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,525 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,625 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:28,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:28,627 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(933): Client=jenkins//136.243.104.168 procedure request for: flush-table-proc 2023-04-28 04:58:28,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 04:58:28,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 04:58:28,628 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 04:58:28,628 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:28,628 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 04:58:28,628 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 04:58:28,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,629 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,640 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,640 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,641 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,641 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 04:58:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 04:58:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 04:58:28,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 04:58:28,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 04:58:28,642 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,642 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. started... 2023-04-28 04:58:28,642 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. as already flushing 2023-04-28 04:58:28,642 DEBUG [rs(jenkins-hbase12.apache.org,37663,1682657733313)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 04:58:28,642 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 04:58:28,645 DEBUG [Thread-3211] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cc4808 to 127.0.0.1:52932 2023-04-28 04:58:28,645 DEBUG [Thread-3211] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:28,646 DEBUG [Thread-3203] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c00a18c to 127.0.0.1:52932 2023-04-28 04:58:28,646 DEBUG [Thread-3205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3da68fe0 to 127.0.0.1:52932 2023-04-28 04:58:28,646 DEBUG [Thread-3203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:28,646 DEBUG [Thread-3205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:28,647 DEBUG [Thread-3209] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e63396 to 127.0.0.1:52932 2023-04-28 04:58:28,648 DEBUG [Thread-3209] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:28,649 DEBUG [Thread-3207] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fd90ac8 to 127.0.0.1:52932 2023-04-28 04:58:28,649 DEBUG [Thread-3207] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:28,651 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,651 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,663 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 04:58:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47380 deadline: 1682657968663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase12.apache.org,37663,1682657733313' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 04:58:28,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c3eef7[Count = 0] remaining members to acquire global barrier 2023-04-28 04:58:28,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 04:58:28,664 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,672 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,672 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 04:58:28,672 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 04:58:28,672 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase12.apache.org,37663,1682657733313' in zk 2023-04-28 04:58:28,672 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,672 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 04:58:28,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 04:58:28,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,683 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,683 DEBUG [member: 'jenkins-hbase12.apache.org,37663,1682657733313' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 04:58:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase12.apache.org,37663,1682657733313': 2023-04-28 04:58:28,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase12.apache.org,37663,1682657733313' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 04:58:28,687 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 04:58:28,687 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 04:58:28,687 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 04:58:28,687 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 04:58:28,687 INFO [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 04:58:28,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657968690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47464 deadline: 1682657968690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657968691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,693 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,693 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,693 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 04:58:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 04:58:28,694 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 04:58:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 04:58:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,696 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 04:58:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 04:58:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 04:58:28,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 04:58:28,704 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 04:58:28,704 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 04:58:28,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 04:58:28,704 DEBUG [(jenkins-hbase12.apache.org,38117,1682657732105)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 04:58:28,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 04:58:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 04:58:28,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 04:58:28,704 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:28,704 DEBUG [Thread-3202] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 04:58:28,705 DEBUG [Thread-3202] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 04:58:28,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 04:58:28,705 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 04:58:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 04:58:28,805 DEBUG [Thread-3202] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 04:58:28,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 04:58:28,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/4dfad2c7eaab48d7a1b9ca2242ffb028 2023-04-28 04:58:29,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47380 deadline: 1682657969167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:29,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:29,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37663] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47410 deadline: 1682657969195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:29,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47464 deadline: 1682657969195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:29,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 04:58:29,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.104.168:47428 deadline: 1682657969196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:29,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1271 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/f4bddee15d244b199b9c1f812d680089 2023-04-28 04:58:29,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/a1f0d02b0a204b15aa1bbd6ddf95fe27 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a1f0d02b0a204b15aa1bbd6ddf95fe27 2023-04-28 04:58:29,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a1f0d02b0a204b15aa1bbd6ddf95fe27, entries=200, sequenceid=1271, filesize=32.3 K 2023-04-28 04:58:29,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/4dfad2c7eaab48d7a1b9ca2242ffb028 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4dfad2c7eaab48d7a1b9ca2242ffb028 2023-04-28 04:58:29,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4dfad2c7eaab48d7a1b9ca2242ffb028, entries=150, sequenceid=1271, filesize=11.9 K 2023-04-28 04:58:29,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/f4bddee15d244b199b9c1f812d680089 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/f4bddee15d244b199b9c1f812d680089 2023-04-28 04:58:29,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/f4bddee15d244b199b9c1f812d680089, entries=150, sequenceid=1271, filesize=11.9 K 2023-04-28 04:58:29,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~156.33 KB/160080, currentSize=127.47 KB/130530 for a6bab1698a9b263a4df9a801810dbac0 in 1267ms, sequenceid=1271, compaction requested=true 2023-04-28 04:58:29,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:29,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:29,303 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:29,303 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:29,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:29,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:29,304 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:29,304 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/A is initiating minor compaction (all files) 2023-04-28 04:58:29,304 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/A in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:29,304 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:29,304 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a434e386affb4811b69ed3d2fe48ff38, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/19647301c9da465ead4d80c9ef641f06, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a1f0d02b0a204b15aa1bbd6ddf95fe27] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=83.8 K 2023-04-28 04:58:29,304 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/B is initiating minor compaction (all files) 2023-04-28 04:58:29,304 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/B in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:29,304 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/110c9f3ff9634237a626cb1c1d5694a6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/317315fe310647d19e0fe3aaea1d2480, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4dfad2c7eaab48d7a1b9ca2242ffb028] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=36.3 K 2023-04-28 04:58:29,305 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting a434e386affb4811b69ed3d2fe48ff38, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1682657905153 2023-04-28 04:58:29,305 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 110c9f3ff9634237a626cb1c1d5694a6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1682657905153 2023-04-28 04:58:29,305 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting 19647301c9da465ead4d80c9ef641f06, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1682657905465 2023-04-28 04:58:29,305 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 317315fe310647d19e0fe3aaea1d2480, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1682657905465 2023-04-28 04:58:29,305 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] compactions.Compactor(207): Compacting a1f0d02b0a204b15aa1bbd6ddf95fe27, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1271, earliestPutTs=1682657908031 2023-04-28 04:58:29,306 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 4dfad2c7eaab48d7a1b9ca2242ffb028, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1271, earliestPutTs=1682657908032 2023-04-28 04:58:29,313 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#B#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:29,314 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#A#compaction#491 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:29,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37663] regionserver.HRegion(9158): Flush requested on a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:29,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 04:58:29,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:29,618 DEBUG [Thread-3194] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a1010a6 to 127.0.0.1:52932 2023-04-28 04:58:29,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:29,618 DEBUG [Thread-3194] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:29,618 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:29,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:29,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:29,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 04:58:29,729 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/14b3010bcb194e5397490a0991c8cb92 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/14b3010bcb194e5397490a0991c8cb92 2023-04-28 04:58:29,734 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/B of a6bab1698a9b263a4df9a801810dbac0 into 14b3010bcb194e5397490a0991c8cb92(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:29,734 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:29,734 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/B, priority=13, startTime=1682657909303; duration=0sec 2023-04-28 04:58:29,734 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 04:58:29,734 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 04:58:29,735 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 04:58:29,735 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1912): a6bab1698a9b263a4df9a801810dbac0/C is initiating minor compaction (all files) 2023-04-28 04:58:29,735 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a6bab1698a9b263a4df9a801810dbac0/C in TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:29,735 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/be821825be594044930b785ddeef0ad5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/752e20441a9a499daecb5d5dff730106, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/f4bddee15d244b199b9c1f812d680089] into tmpdir=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp, totalSize=36.3 K 2023-04-28 04:58:29,736 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting be821825be594044930b785ddeef0ad5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1682657905153 2023-04-28 04:58:29,736 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting 752e20441a9a499daecb5d5dff730106, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1682657905465 2023-04-28 04:58:29,736 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] compactions.Compactor(207): Compacting f4bddee15d244b199b9c1f812d680089, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1271, earliestPutTs=1682657908032 2023-04-28 04:58:29,742 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] throttle.PressureAwareThroughputController(145): a6bab1698a9b263a4df9a801810dbac0#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 04:58:30,040 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042898c7af4eb39a4a918331f1d57f591b28 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042898c7af4eb39a4a918331f1d57f591b28 2023-04-28 04:58:30,157 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/a38de3f9280f4b34aad72e941241cd6b as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a38de3f9280f4b34aad72e941241cd6b 2023-04-28 04:58:30,163 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/C of a6bab1698a9b263a4df9a801810dbac0 into a38de3f9280f4b34aad72e941241cd6b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 04:58:30,163 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:30,163 INFO [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/C, priority=13, startTime=1682657909303; duration=0sec 2023-04-28 04:58:30,163 DEBUG [RS:0;jenkins-hbase12:37663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:30,170 DEBUG [Thread-3192] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f11e84a to 127.0.0.1:52932 2023-04-28 04:58:30,170 DEBUG [Thread-3192] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:30,204 DEBUG [Thread-3200] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078cd331 to 127.0.0.1:52932 2023-04-28 04:58:30,204 DEBUG [Thread-3200] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:30,206 DEBUG [Thread-3196] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4143b9ae to 127.0.0.1:52932 2023-04-28 04:58:30,206 DEBUG [Thread-3196] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:30,207 DEBUG [Thread-3198] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x686af57e to 127.0.0.1:52932 2023-04-28 04:58:30,207 DEBUG [Thread-3198] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 309 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 181 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 251 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 232 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(394): wrote 266 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5347 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5447 2023-04-28 04:58:30,207 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5346 2023-04-28 04:58:30,208 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5330 2023-04-28 04:58:30,208 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(398): read 5459 2023-04-28 04:58:30,208 INFO [Listener at localhost.localdomain/36983] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 04:58:30,208 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:58:30,208 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5be55982 to 127.0.0.1:52932 2023-04-28 04:58:30,208 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:30,208 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 04:58:30,209 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$11(2418): Client=jenkins//136.243.104.168 disable TestAcidGuarantees 2023-04-28 04:58:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:30,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:30,212 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657910212"}]},"ts":"1682657910212"} 2023-04-28 04:58:30,213 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 04:58:30,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:30,332 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 04:58:30,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, UNASSIGN}] 2023-04-28 04:58:30,336 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, UNASSIGN 2023-04-28 04:58:30,337 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=CLOSING, regionLocation=jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:30,337 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657910336"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682657910336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682657910336"}]},"ts":"1682657910336"} 2023-04-28 04:58:30,338 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313}] 2023-04-28 04:58:30,449 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1308, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/1f7162a9b61940c085f77005cc763015 2023-04-28 04:58:30,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(109): Close a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:30,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing a6bab1698a9b263a4df9a801810dbac0, disabling compactions & flushes 2023-04-28 04:58:30,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:30,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:30,541 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/fe455791884d4bbd98894c25ad1efa56 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe455791884d4bbd98894c25ad1efa56 2023-04-28 04:58:30,544 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a6bab1698a9b263a4df9a801810dbac0/A of a6bab1698a9b263a4df9a801810dbac0 into fe455791884d4bbd98894c25ad1efa56(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 04:58:30,544 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:30,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:30,545 INFO [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0., storeName=a6bab1698a9b263a4df9a801810dbac0/A, priority=13, startTime=1682657909303; duration=1sec 2023-04-28 04:58:30,545 DEBUG [RS:0;jenkins-hbase12:37663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 04:58:30,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:30,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1308 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/e6c06bc2d9154e39bfc17662d93b126e 2023-04-28 04:58:31,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1308 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/07b8e39619314d0eb74be921a8805bc8 2023-04-28 04:58:31,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/1f7162a9b61940c085f77005cc763015 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/1f7162a9b61940c085f77005cc763015 2023-04-28 04:58:31,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/1f7162a9b61940c085f77005cc763015, entries=150, sequenceid=1308, filesize=25.4 K 2023-04-28 04:58:31,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/e6c06bc2d9154e39bfc17662d93b126e as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/e6c06bc2d9154e39bfc17662d93b126e 2023-04-28 04:58:31,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/e6c06bc2d9154e39bfc17662d93b126e, entries=150, sequenceid=1308, filesize=11.9 K 2023-04-28 04:58:31,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/07b8e39619314d0eb74be921a8805bc8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/07b8e39619314d0eb74be921a8805bc8 2023-04-28 04:58:31,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/07b8e39619314d0eb74be921a8805bc8, entries=150, sequenceid=1308, filesize=11.9 K 2023-04-28 04:58:31,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=26.84 KB/27480 for a6bab1698a9b263a4df9a801810dbac0 in 1684ms, sequenceid=1308, compaction requested=false 2023-04-28 04:58:31,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:31,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. after waiting 0 ms 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:31,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing a6bab1698a9b263a4df9a801810dbac0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=A 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=B 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a6bab1698a9b263a4df9a801810dbac0, store=C 2023-04-28 04:58:31,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 04:58:31,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:31,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428cbfd7ea232984179b7c78007e7bd88c1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cbfd7ea232984179b7c78007e7bd88c1 2023-04-28 04:58:32,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1318, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/d422450cf85b4aa3abe975c32dd0aa48 2023-04-28 04:58:32,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:32,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1318 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/0753747cd5794432996d0cb8a9f48cc3 2023-04-28 04:58:32,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1318 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/c2e4398b28cd4777801d0c9238f75980 2023-04-28 04:58:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/A/d422450cf85b4aa3abe975c32dd0aa48 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/d422450cf85b4aa3abe975c32dd0aa48 2023-04-28 04:58:32,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/d422450cf85b4aa3abe975c32dd0aa48, entries=50, sequenceid=1318, filesize=11.7 K 2023-04-28 04:58:32,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/B/0753747cd5794432996d0cb8a9f48cc3 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/0753747cd5794432996d0cb8a9f48cc3 2023-04-28 04:58:32,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/0753747cd5794432996d0cb8a9f48cc3, entries=50, sequenceid=1318, filesize=7.1 K 2023-04-28 04:58:32,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/.tmp/C/c2e4398b28cd4777801d0c9238f75980 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c2e4398b28cd4777801d0c9238f75980 2023-04-28 04:58:32,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c2e4398b28cd4777801d0c9238f75980, entries=50, sequenceid=1318, filesize=7.1 K 2023-04-28 04:58:32,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a6bab1698a9b263a4df9a801810dbac0 in 1269ms, sequenceid=1318, compaction requested=true 2023-04-28 04:58:32,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/2e1f1e95885c4db7bb74481d1cbb507a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/ec572e3092624ff097b186e52d55931b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/eba74652b574409db185f53527ff52f6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe52281201c0435c90072c196f7206c1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/348c49e51d3443ed9ac5eac0e777c98d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/e5e0944e689b40eda75a9469e0a8b0f8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/c2106e2ed4784e30b196f313002b0c85, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/4da84585b32244a5af85281029e03066, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/08e87db2611a419da023f9d0766886cd, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/11515da1bdc7421395522dde6b85be9a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/05ad320028844df2b415b185fdfd8696, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/552b6c4fa8e64297b1be7fa472b68d8f, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/38ba2c6b21d34b68868f51f8c6a89761, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/791a2fc5b884455dacc202721d43dd23, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6bec7cf5e8e14dd1b5835a005af62d2b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/5a3264cb1bc04116a172928d388f5c1e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6884d6ced2d7466294f0bb29b8a2ee19, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/7b7d01c9bb3b45dd937b7e84381a56e5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/f0401ebed23442feb3e65f4fe28bce7b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a434e386affb4811b69ed3d2fe48ff38, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/19647301c9da465ead4d80c9ef641f06, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a1f0d02b0a204b15aa1bbd6ddf95fe27] to archive 2023-04-28 04:58:32,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:58:32,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/2e1f1e95885c4db7bb74481d1cbb507a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/2e1f1e95885c4db7bb74481d1cbb507a 2023-04-28 04:58:32,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/ec572e3092624ff097b186e52d55931b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/ec572e3092624ff097b186e52d55931b 2023-04-28 04:58:32,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/eba74652b574409db185f53527ff52f6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/eba74652b574409db185f53527ff52f6 2023-04-28 04:58:32,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe52281201c0435c90072c196f7206c1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe52281201c0435c90072c196f7206c1 2023-04-28 04:58:32,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/348c49e51d3443ed9ac5eac0e777c98d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/348c49e51d3443ed9ac5eac0e777c98d 2023-04-28 04:58:32,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/e5e0944e689b40eda75a9469e0a8b0f8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/e5e0944e689b40eda75a9469e0a8b0f8 2023-04-28 04:58:32,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/c2106e2ed4784e30b196f313002b0c85 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/c2106e2ed4784e30b196f313002b0c85 2023-04-28 04:58:32,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/4da84585b32244a5af85281029e03066 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/4da84585b32244a5af85281029e03066 2023-04-28 04:58:32,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/08e87db2611a419da023f9d0766886cd to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/08e87db2611a419da023f9d0766886cd 2023-04-28 04:58:32,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/11515da1bdc7421395522dde6b85be9a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/11515da1bdc7421395522dde6b85be9a 2023-04-28 04:58:32,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/05ad320028844df2b415b185fdfd8696 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/05ad320028844df2b415b185fdfd8696 2023-04-28 04:58:32,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/552b6c4fa8e64297b1be7fa472b68d8f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/552b6c4fa8e64297b1be7fa472b68d8f 2023-04-28 04:58:32,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/38ba2c6b21d34b68868f51f8c6a89761 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/38ba2c6b21d34b68868f51f8c6a89761 2023-04-28 04:58:32,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/791a2fc5b884455dacc202721d43dd23 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/791a2fc5b884455dacc202721d43dd23 2023-04-28 04:58:32,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6bec7cf5e8e14dd1b5835a005af62d2b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6bec7cf5e8e14dd1b5835a005af62d2b 2023-04-28 04:58:32,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/5a3264cb1bc04116a172928d388f5c1e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/5a3264cb1bc04116a172928d388f5c1e 2023-04-28 04:58:32,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6884d6ced2d7466294f0bb29b8a2ee19 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/6884d6ced2d7466294f0bb29b8a2ee19 2023-04-28 04:58:32,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/7b7d01c9bb3b45dd937b7e84381a56e5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/7b7d01c9bb3b45dd937b7e84381a56e5 2023-04-28 04:58:32,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/f0401ebed23442feb3e65f4fe28bce7b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/f0401ebed23442feb3e65f4fe28bce7b 2023-04-28 04:58:32,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a434e386affb4811b69ed3d2fe48ff38 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a434e386affb4811b69ed3d2fe48ff38 2023-04-28 04:58:32,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/19647301c9da465ead4d80c9ef641f06 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/19647301c9da465ead4d80c9ef641f06 2023-04-28 04:58:32,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a1f0d02b0a204b15aa1bbd6ddf95fe27 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/a1f0d02b0a204b15aa1bbd6ddf95fe27 2023-04-28 04:58:32,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/05426c38bc9d4536be7b3cad3387a873, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/951d968c493c4cd7a2114103ef5bb707, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b7cc1230acc0497fa438abdaceeb2f6d, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/241457af47d3401c97bc5a3e2724511c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/55d66d95370d4d1a8dedde7cc1e9d1d5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/89d517f02ffc426e857a5a3e43f9cdd9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/d2a540177603425e819b93cfda601478, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/64ba3a702bc94e889c67376544a14179, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b86d91e4393d455995d6eb910321e8f3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/ebf9cedb71534e958d0de90e8b6288b9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4a7e7943b9b74b3fbaa0e3d982b695a9, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/9c2d1681e1054c8c856f74110082b3a1, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/c4cace55308f4722b21d2056c29ef14a, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b2811535568e47e8920b4712a513d290, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/3ab3d54c985741b0825c3d72a1de881c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/089f0a5534f24115b3c1883a82efd7e4, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/fe0153d478f84467a1496dbbaf8047a7, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/7cc73c10da7c433ba27737e0350279c2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/110c9f3ff9634237a626cb1c1d5694a6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/58129b7d7aca49699950922c1aaec3b5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/317315fe310647d19e0fe3aaea1d2480, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4dfad2c7eaab48d7a1b9ca2242ffb028] to archive 2023-04-28 04:58:32,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:58:32,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/05426c38bc9d4536be7b3cad3387a873 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/05426c38bc9d4536be7b3cad3387a873 2023-04-28 04:58:32,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/951d968c493c4cd7a2114103ef5bb707 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/951d968c493c4cd7a2114103ef5bb707 2023-04-28 04:58:32,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b7cc1230acc0497fa438abdaceeb2f6d to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b7cc1230acc0497fa438abdaceeb2f6d 2023-04-28 04:58:32,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/241457af47d3401c97bc5a3e2724511c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/241457af47d3401c97bc5a3e2724511c 2023-04-28 04:58:32,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/55d66d95370d4d1a8dedde7cc1e9d1d5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/55d66d95370d4d1a8dedde7cc1e9d1d5 2023-04-28 04:58:32,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/89d517f02ffc426e857a5a3e43f9cdd9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/89d517f02ffc426e857a5a3e43f9cdd9 2023-04-28 04:58:32,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/d2a540177603425e819b93cfda601478 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/d2a540177603425e819b93cfda601478 2023-04-28 04:58:32,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/64ba3a702bc94e889c67376544a14179 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/64ba3a702bc94e889c67376544a14179 2023-04-28 04:58:32,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b86d91e4393d455995d6eb910321e8f3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b86d91e4393d455995d6eb910321e8f3 2023-04-28 04:58:32,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/ebf9cedb71534e958d0de90e8b6288b9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/ebf9cedb71534e958d0de90e8b6288b9 2023-04-28 04:58:32,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4a7e7943b9b74b3fbaa0e3d982b695a9 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4a7e7943b9b74b3fbaa0e3d982b695a9 2023-04-28 04:58:32,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/9c2d1681e1054c8c856f74110082b3a1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/9c2d1681e1054c8c856f74110082b3a1 2023-04-28 04:58:32,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/c4cace55308f4722b21d2056c29ef14a to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/c4cace55308f4722b21d2056c29ef14a 2023-04-28 04:58:32,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b2811535568e47e8920b4712a513d290 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/b2811535568e47e8920b4712a513d290 2023-04-28 04:58:32,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/3ab3d54c985741b0825c3d72a1de881c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/3ab3d54c985741b0825c3d72a1de881c 2023-04-28 04:58:32,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/089f0a5534f24115b3c1883a82efd7e4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/089f0a5534f24115b3c1883a82efd7e4 2023-04-28 04:58:32,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/fe0153d478f84467a1496dbbaf8047a7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/fe0153d478f84467a1496dbbaf8047a7 2023-04-28 04:58:32,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/7cc73c10da7c433ba27737e0350279c2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/7cc73c10da7c433ba27737e0350279c2 2023-04-28 04:58:32,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/110c9f3ff9634237a626cb1c1d5694a6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/110c9f3ff9634237a626cb1c1d5694a6 2023-04-28 04:58:32,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/58129b7d7aca49699950922c1aaec3b5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/58129b7d7aca49699950922c1aaec3b5 2023-04-28 04:58:32,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/317315fe310647d19e0fe3aaea1d2480 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/317315fe310647d19e0fe3aaea1d2480 2023-04-28 04:58:32,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4dfad2c7eaab48d7a1b9ca2242ffb028 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/4dfad2c7eaab48d7a1b9ca2242ffb028 2023-04-28 04:58:32,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9f73f635600b476cb43724f7552f712e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a098cfa715144c25a96001777ca3fac0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5ba4dd76f97c49758f0ee773dfd53774, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/1e7d15b30f134effa0fc8363ea85f968, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c711fa60ec954051b65e0f97e149c6f2, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/74daae97538e4d6391e93c9f7db14bef, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/03a4e99560544c299a883d7931d618e3, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9872699a3aaa41219e5f3e75cd8d57c8, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5302371a9bfc417f8ee513188d193e66, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/e356bc5a5f08421da7b443777b4c0c5e, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/7f74c9d8acd54695a85f22b0ce7fae6b, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5f7df0822f36431c97d7da32fb83e46c, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/d1db41167f3d4160b97fe1ee6486c5a0, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a99d215e5e044983b5f90fe6e62f4279, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/0e044f3c98204bf9ae743fe53bacbecc, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/ee8a466914fb4e6ba32b792f8b479977, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/cf8695a8bedc47fc90f39c88bdd34343, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/de2d03b12e7e4aeca505ff2a11b989a6, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/be821825be594044930b785ddeef0ad5, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a8bda8afb53b4b2f907e279ccb613b89, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/752e20441a9a499daecb5d5dff730106, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/f4bddee15d244b199b9c1f812d680089] to archive 2023-04-28 04:58:32,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 04:58:32,633 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9f73f635600b476cb43724f7552f712e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9f73f635600b476cb43724f7552f712e 2023-04-28 04:58:32,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a098cfa715144c25a96001777ca3fac0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a098cfa715144c25a96001777ca3fac0 2023-04-28 04:58:32,635 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5ba4dd76f97c49758f0ee773dfd53774 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5ba4dd76f97c49758f0ee773dfd53774 2023-04-28 04:58:32,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/1e7d15b30f134effa0fc8363ea85f968 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/1e7d15b30f134effa0fc8363ea85f968 2023-04-28 04:58:32,637 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c711fa60ec954051b65e0f97e149c6f2 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c711fa60ec954051b65e0f97e149c6f2 2023-04-28 04:58:32,638 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/74daae97538e4d6391e93c9f7db14bef to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/74daae97538e4d6391e93c9f7db14bef 2023-04-28 04:58:32,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/03a4e99560544c299a883d7931d618e3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/03a4e99560544c299a883d7931d618e3 2023-04-28 04:58:32,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9872699a3aaa41219e5f3e75cd8d57c8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/9872699a3aaa41219e5f3e75cd8d57c8 2023-04-28 04:58:32,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5302371a9bfc417f8ee513188d193e66 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5302371a9bfc417f8ee513188d193e66 2023-04-28 04:58:32,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/e356bc5a5f08421da7b443777b4c0c5e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/e356bc5a5f08421da7b443777b4c0c5e 2023-04-28 04:58:32,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/7f74c9d8acd54695a85f22b0ce7fae6b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/7f74c9d8acd54695a85f22b0ce7fae6b 2023-04-28 04:58:32,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5f7df0822f36431c97d7da32fb83e46c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/5f7df0822f36431c97d7da32fb83e46c 2023-04-28 04:58:32,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/d1db41167f3d4160b97fe1ee6486c5a0 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/d1db41167f3d4160b97fe1ee6486c5a0 2023-04-28 04:58:32,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a99d215e5e044983b5f90fe6e62f4279 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a99d215e5e044983b5f90fe6e62f4279 2023-04-28 04:58:32,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/0e044f3c98204bf9ae743fe53bacbecc to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/0e044f3c98204bf9ae743fe53bacbecc 2023-04-28 04:58:32,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/ee8a466914fb4e6ba32b792f8b479977 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/ee8a466914fb4e6ba32b792f8b479977 2023-04-28 04:58:32,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/cf8695a8bedc47fc90f39c88bdd34343 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/cf8695a8bedc47fc90f39c88bdd34343 2023-04-28 04:58:32,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/de2d03b12e7e4aeca505ff2a11b989a6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/de2d03b12e7e4aeca505ff2a11b989a6 2023-04-28 04:58:32,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/be821825be594044930b785ddeef0ad5 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/be821825be594044930b785ddeef0ad5 2023-04-28 04:58:32,648 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a8bda8afb53b4b2f907e279ccb613b89 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a8bda8afb53b4b2f907e279ccb613b89 2023-04-28 04:58:32,649 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/752e20441a9a499daecb5d5dff730106 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/752e20441a9a499daecb5d5dff730106 2023-04-28 04:58:32,649 DEBUG [StoreCloser-TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/f4bddee15d244b199b9c1f812d680089 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/f4bddee15d244b199b9c1f812d680089 2023-04-28 04:58:32,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/recovered.edits/1321.seqid, newMaxSeqId=1321, maxSeqId=4 2023-04-28 04:58:32,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0. 2023-04-28 04:58:32,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for a6bab1698a9b263a4df9a801810dbac0: 2023-04-28 04:58:32,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.UnassignRegionHandler(147): Closed a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:32,654 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a6bab1698a9b263a4df9a801810dbac0, regionState=CLOSED 2023-04-28 04:58:32,654 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682657912654"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682657912654"}]},"ts":"1682657912654"} 2023-04-28 04:58:32,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-28 04:58:32,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure a6bab1698a9b263a4df9a801810dbac0, server=jenkins-hbase12.apache.org,37663,1682657733313 in 2.3170 sec 2023-04-28 04:58:32,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-28 04:58:32,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a6bab1698a9b263a4df9a801810dbac0, UNASSIGN in 2.3230 sec 2023-04-28 04:58:32,657 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682657912657"}]},"ts":"1682657912657"} 2023-04-28 04:58:32,660 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 04:58:32,769 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 04:58:32,771 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5610 sec 2023-04-28 04:58:34,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 04:58:34,320 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-28 04:58:34,321 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.HMaster$5(2228): Client=jenkins//136.243.104.168 delete TestAcidGuarantees 2023-04-28 04:58:34,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:34,325 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:34,326 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:34,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 04:58:34,333 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:34,338 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C, FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/recovered.edits] 2023-04-28 04:58:34,343 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/1f7162a9b61940c085f77005cc763015 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/1f7162a9b61940c085f77005cc763015 2023-04-28 04:58:34,345 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/d422450cf85b4aa3abe975c32dd0aa48 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/d422450cf85b4aa3abe975c32dd0aa48 2023-04-28 04:58:34,347 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe455791884d4bbd98894c25ad1efa56 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/A/fe455791884d4bbd98894c25ad1efa56 2023-04-28 04:58:34,351 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/0753747cd5794432996d0cb8a9f48cc3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/0753747cd5794432996d0cb8a9f48cc3 2023-04-28 04:58:34,354 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/14b3010bcb194e5397490a0991c8cb92 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/14b3010bcb194e5397490a0991c8cb92 2023-04-28 04:58:34,355 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/e6c06bc2d9154e39bfc17662d93b126e to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/B/e6c06bc2d9154e39bfc17662d93b126e 2023-04-28 04:58:34,357 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/07b8e39619314d0eb74be921a8805bc8 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/07b8e39619314d0eb74be921a8805bc8 2023-04-28 04:58:34,358 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a38de3f9280f4b34aad72e941241cd6b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/a38de3f9280f4b34aad72e941241cd6b 2023-04-28 04:58:34,359 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c2e4398b28cd4777801d0c9238f75980 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/C/c2e4398b28cd4777801d0c9238f75980 2023-04-28 04:58:34,360 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/recovered.edits/1321.seqid to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0/recovered.edits/1321.seqid 2023-04-28 04:58:34,361 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/.tmp/data/default/TestAcidGuarantees/a6bab1698a9b263a4df9a801810dbac0 2023-04-28 04:58:34,361 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 04:58:34,361 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 04:58:34,362 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 04:58:34,364 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042805cfed0e61834009aa2d08475f4e956f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042805cfed0e61834009aa2d08475f4e956f 2023-04-28 04:58:34,365 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c7bc51c6f6641d09b1e323686fe486b to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280c7bc51c6f6641d09b1e323686fe486b 2023-04-28 04:58:34,366 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042834ce01d826dc4c309d4e4f17d7b4e901 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042834ce01d826dc4c309d4e4f17d7b4e901 2023-04-28 04:58:34,367 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286f7bfa3ec5a7443db5200e13d3e3cfda to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286f7bfa3ec5a7443db5200e13d3e3cfda 2023-04-28 04:58:34,368 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286f8740845bb846d89e273db7ff78a586 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286f8740845bb846d89e273db7ff78a586 2023-04-28 04:58:34,369 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042879cf396bbe664fbeb659121ff05a13b3 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042879cf396bbe664fbeb659121ff05a13b3 2023-04-28 04:58:34,370 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428871743296881476f9d91e995d2ca0c38 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428871743296881476f9d91e995d2ca0c38 2023-04-28 04:58:34,371 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428902b33ea63834736a4917ca732139199 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428902b33ea63834736a4917ca732139199 2023-04-28 04:58:34,373 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042895e44f74dd62484eb6f1cc38863bfef6 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042895e44f74dd62484eb6f1cc38863bfef6 2023-04-28 04:58:34,374 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042898c7af4eb39a4a918331f1d57f591b28 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042898c7af4eb39a4a918331f1d57f591b28 2023-04-28 04:58:34,375 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ad7d3d4836c426b8ebfa2ec167b2c3c to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289ad7d3d4836c426b8ebfa2ec167b2c3c 2023-04-28 04:58:34,376 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aac4042054c44bb6880996a15ade802f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aac4042054c44bb6880996a15ade802f 2023-04-28 04:58:34,377 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c4808a34e4b74ab4aeb945d8e113f357 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c4808a34e4b74ab4aeb945d8e113f357 2023-04-28 04:58:34,378 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c9ab39717f6a47d4b2ca8862002cada1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c9ab39717f6a47d4b2ca8862002cada1 2023-04-28 04:58:34,380 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cbfd7ea232984179b7c78007e7bd88c1 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428cbfd7ea232984179b7c78007e7bd88c1 2023-04-28 04:58:34,381 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dc52512e932c4548b9dc5cbdc5397ca7 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dc52512e932c4548b9dc5cbdc5397ca7 2023-04-28 04:58:34,382 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dcb6499fcfd54e31a5c395bdeb4cc52f to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dcb6499fcfd54e31a5c395bdeb4cc52f 2023-04-28 04:58:34,384 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428df6d731c2611407f87c887ce01109a74 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428df6d731c2611407f87c887ce01109a74 2023-04-28 04:58:34,385 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ed1008072e6d4d448f98c801ae9416d4 to hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ed1008072e6d4d448f98c801ae9416d4 2023-04-28 04:58:34,386 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 04:58:34,388 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:34,390 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 04:58:34,392 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 04:58:34,394 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:34,394 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 04:58:34,394 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682657914394"}]},"ts":"9223372036854775807"} 2023-04-28 04:58:34,396 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 04:58:34,396 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a6bab1698a9b263a4df9a801810dbac0, NAME => 'TestAcidGuarantees,,1682657885199.a6bab1698a9b263a4df9a801810dbac0.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 04:58:34,397 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 04:58:34,397 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682657914397"}]},"ts":"9223372036854775807"} 2023-04-28 04:58:34,399 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 04:58:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 04:58:34,469 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 04:58:34,470 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 148 msec 2023-04-28 04:58:34,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38117] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 04:58:34,632 INFO [Listener at localhost.localdomain/36983] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-28 04:58:34,649 INFO [Listener at localhost.localdomain/36983] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 228), OpenFileDescriptor=545 (was 562), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=461 (was 505), ProcessCount=171 (was 173), AvailableMemoryMB=10205 (was 10275) 2023-04-28 04:58:34,649 INFO [Listener at localhost.localdomain/36983] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-28 04:58:34,650 INFO [Listener at localhost.localdomain/36983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 04:58:34,650 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c8a8f6d to 127.0.0.1:52932 2023-04-28 04:58:34,650 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:34,650 DEBUG [Listener at localhost.localdomain/36983] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-28 04:58:34,650 DEBUG [Listener at localhost.localdomain/36983] util.JVMClusterUtil(257): Found active master hash=376958400, stopped=false 2023-04-28 04:58:34,650 INFO [Listener at localhost.localdomain/36983] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:58:34,672 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-28 04:58:34,672 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-28 04:58:34,672 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:58:34,672 INFO [Listener at localhost.localdomain/36983] procedure2.ProcedureExecutor(629): Stopping 2023-04-28 04:58:34,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 04:58:34,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 04:58:34,673 DEBUG [Listener at localhost.localdomain/36983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cf0e276 to 127.0.0.1:52932 2023-04-28 04:58:34,673 DEBUG [Listener at localhost.localdomain/36983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:34,673 INFO [Listener at localhost.localdomain/36983] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase12.apache.org,37663,1682657733313' ***** 2023-04-28 04:58:34,673 INFO [Listener at localhost.localdomain/36983] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-28 04:58:34,674 INFO [RS:0;jenkins-hbase12:37663] regionserver.HeapMemoryManager(220): Stopping 2023-04-28 04:58:34,674 INFO [RS:0;jenkins-hbase12:37663] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-28 04:58:34,674 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-28 04:58:34,674 INFO [RS:0;jenkins-hbase12:37663] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-28 04:58:34,675 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(3304): Received CLOSE for b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:58:34,676 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1145): stopping server jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:34,676 DEBUG [RS:0;jenkins-hbase12:37663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02dca4c8 to 127.0.0.1:52932 2023-04-28 04:58:34,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing b7a9f863068c5f23d85b7830bfb4ddae, disabling compactions & flushes 2023-04-28 04:58:34,677 DEBUG [RS:0;jenkins-hbase12:37663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:34,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:58:34,678 INFO [RS:0;jenkins-hbase12:37663] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-28 04:58:34,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:58:34,678 INFO [RS:0;jenkins-hbase12:37663] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-28 04:58:34,678 INFO [RS:0;jenkins-hbase12:37663] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-28 04:58:34,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. after waiting 0 ms 2023-04-28 04:58:34,678 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-28 04:58:34,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:58:34,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing b7a9f863068c5f23d85b7830bfb4ddae 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-28 04:58:34,678 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-28 04:58:34,679 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1479): Online Regions={b7a9f863068c5f23d85b7830bfb4ddae=hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae., 1588230740=hbase:meta,,1.1588230740} 2023-04-28 04:58:34,679 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-28 04:58:34,680 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-28 04:58:34,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-28 04:58:34,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-28 04:58:34,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-28 04:58:34,680 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-28 04:58:34,682 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740, b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:58:34,731 INFO [regionserver/jenkins-hbase12:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-28 04:58:34,882 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740, b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:58:34,982 INFO [regionserver/jenkins-hbase12:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-28 04:58:34,982 INFO [regionserver/jenkins-hbase12:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-28 04:58:35,083 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740, b7a9f863068c5f23d85b7830bfb4ddae 2023-04-28 04:58:35,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/.tmp/info/eeaaaa846923458f923ed2eba01fffdd 2023-04-28 04:58:35,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/.tmp/info/f3f2fbb273b3454eac793a358be72902 2023-04-28 04:58:35,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/.tmp/info/eeaaaa846923458f923ed2eba01fffdd as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/info/eeaaaa846923458f923ed2eba01fffdd 2023-04-28 04:58:35,101 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f3f2fbb273b3454eac793a358be72902 2023-04-28 04:58:35,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/info/eeaaaa846923458f923ed2eba01fffdd, entries=2, sequenceid=6, filesize=4.8 K 2023-04-28 04:58:35,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for b7a9f863068c5f23d85b7830bfb4ddae in 427ms, sequenceid=6, compaction requested=false 2023-04-28 04:58:35,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/namespace/b7a9f863068c5f23d85b7830bfb4ddae/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-28 04:58:35,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:58:35,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for b7a9f863068c5f23d85b7830bfb4ddae: 2023-04-28 04:58:35,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase12:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682657736392.b7a9f863068c5f23d85b7830bfb4ddae. 2023-04-28 04:58:35,283 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 04:58:35,484 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 04:58:35,508 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/.tmp/rep_barrier/2a3d5fa3243f4bf5ba57e84c04f4d52a 2023-04-28 04:58:35,515 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2a3d5fa3243f4bf5ba57e84c04f4d52a 2023-04-28 04:58:35,684 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-28 04:58:35,684 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-28 04:58:35,685 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 04:58:35,885 DEBUG [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 04:58:35,928 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/.tmp/table/f7ab5087bcf3469d9ece3ff37792e823 2023-04-28 04:58:35,938 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f7ab5087bcf3469d9ece3ff37792e823 2023-04-28 04:58:35,939 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/.tmp/info/f3f2fbb273b3454eac793a358be72902 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/info/f3f2fbb273b3454eac793a358be72902 2023-04-28 04:58:35,944 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f3f2fbb273b3454eac793a358be72902 2023-04-28 04:58:35,944 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/info/f3f2fbb273b3454eac793a358be72902, entries=22, sequenceid=93, filesize=7.1 K 2023-04-28 04:58:35,946 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/.tmp/rep_barrier/2a3d5fa3243f4bf5ba57e84c04f4d52a as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/rep_barrier/2a3d5fa3243f4bf5ba57e84c04f4d52a 2023-04-28 04:58:35,951 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2a3d5fa3243f4bf5ba57e84c04f4d52a 2023-04-28 04:58:35,951 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/rep_barrier/2a3d5fa3243f4bf5ba57e84c04f4d52a, entries=6, sequenceid=93, filesize=5.5 K 2023-04-28 04:58:35,951 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/.tmp/table/f7ab5087bcf3469d9ece3ff37792e823 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/table/f7ab5087bcf3469d9ece3ff37792e823 2023-04-28 04:58:35,955 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f7ab5087bcf3469d9ece3ff37792e823 2023-04-28 04:58:35,955 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/table/f7ab5087bcf3469d9ece3ff37792e823, entries=9, sequenceid=93, filesize=5.5 K 2023-04-28 04:58:35,956 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1276ms, sequenceid=93, compaction requested=false 2023-04-28 04:58:35,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-28 04:58:35,965 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-28 04:58:35,966 INFO [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-28 04:58:35,966 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-28 04:58:35,966 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase12:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-28 04:58:36,085 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1171): stopping server jenkins-hbase12.apache.org,37663,1682657733313; all regions closed. 2023-04-28 04:58:36,101 DEBUG [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/oldWALs 2023-04-28 04:58:36,101 INFO [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase12.apache.org%2C37663%2C1682657733313.meta:.meta(num 1682657736044) 2023-04-28 04:58:36,105 DEBUG [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/oldWALs 2023-04-28 04:58:36,105 INFO [RS:0;jenkins-hbase12:37663] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase12.apache.org%2C37663%2C1682657733313:(num 1682657735079) 2023-04-28 04:58:36,105 DEBUG [RS:0;jenkins-hbase12:37663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:36,105 INFO [RS:0;jenkins-hbase12:37663] regionserver.LeaseManager(133): Closed leases 2023-04-28 04:58:36,106 INFO [RS:0;jenkins-hbase12:37663] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase12:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-28 04:58:36,106 INFO [regionserver/jenkins-hbase12:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-28 04:58:36,107 INFO [RS:0;jenkins-hbase12:37663] ipc.NettyRpcServer(158): Stopping server on /136.243.104.168:37663 2023-04-28 04:58:36,216 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 04:58:36,216 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase12.apache.org,37663,1682657733313 2023-04-28 04:58:36,217 ERROR [Listener at localhost.localdomain/36983-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3ace5bcf rejected from java.util.concurrent.ThreadPoolExecutor@3ef1d84e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1785] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 04:58:36,219 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 04:58:36,219 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase12.apache.org,37663,1682657733313] 2023-04-28 04:58:36,219 ERROR [Listener at localhost.localdomain/36983-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@29a8df7 rejected from java.util.concurrent.ThreadPoolExecutor@3ef1d84e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1785] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 04:58:36,220 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase12.apache.org,37663,1682657733313; numProcessing=1 2023-04-28 04:58:36,240 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase12.apache.org,37663,1682657733313 already deleted, retry=false 2023-04-28 04:58:36,240 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase12.apache.org,37663,1682657733313 expired; onlineServers=0 2023-04-28 04:58:36,240 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase12.apache.org,38117,1682657732105' ***** 2023-04-28 04:58:36,240 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-28 04:58:36,241 DEBUG [M:0;jenkins-hbase12:38117] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574ae7fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase12.apache.org/136.243.104.168:0 2023-04-28 04:58:36,241 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegionServer(1145): stopping server jenkins-hbase12.apache.org,38117,1682657732105 2023-04-28 04:58:36,241 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegionServer(1171): stopping server jenkins-hbase12.apache.org,38117,1682657732105; all regions closed. 2023-04-28 04:58:36,241 DEBUG [M:0;jenkins-hbase12:38117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 04:58:36,241 DEBUG [M:0;jenkins-hbase12:38117] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-28 04:58:36,241 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-28 04:58:36,241 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster-HFileCleaner.small.0-1682657734743] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase12:0:becomeActiveMaster-HFileCleaner.small.0-1682657734743,5,FailOnTimeoutGroup] 2023-04-28 04:58:36,241 DEBUG [master/jenkins-hbase12:0:becomeActiveMaster-HFileCleaner.large.0-1682657734742] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase12:0:becomeActiveMaster-HFileCleaner.large.0-1682657734742,5,FailOnTimeoutGroup] 2023-04-28 04:58:36,241 DEBUG [M:0;jenkins-hbase12:38117] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-28 04:58:36,242 INFO [M:0;jenkins-hbase12:38117] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-28 04:58:36,242 INFO [M:0;jenkins-hbase12:38117] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-28 04:58:36,242 INFO [M:0;jenkins-hbase12:38117] hbase.ChoreService(369): Chore service for: master/jenkins-hbase12:0 had [] on shutdown 2023-04-28 04:58:36,242 DEBUG [M:0;jenkins-hbase12:38117] master.HMaster(1512): Stopping service threads 2023-04-28 04:58:36,243 INFO [M:0;jenkins-hbase12:38117] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-28 04:58:36,243 ERROR [M:0;jenkins-hbase12:38117] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-28 04:58:36,244 INFO [M:0;jenkins-hbase12:38117] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-28 04:58:36,244 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-28 04:58:36,251 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-28 04:58:36,251 DEBUG [M:0;jenkins-hbase12:38117] zookeeper.ZKUtil(398): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-28 04:58:36,251 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 04:58:36,251 WARN [M:0;jenkins-hbase12:38117] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-28 04:58:36,251 INFO [M:0;jenkins-hbase12:38117] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-28 04:58:36,251 INFO [M:0;jenkins-hbase12:38117] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-28 04:58:36,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 04:58:36,252 DEBUG [M:0;jenkins-hbase12:38117] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-28 04:58:36,252 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:58:36,252 DEBUG [M:0;jenkins-hbase12:38117] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:58:36,252 DEBUG [M:0;jenkins-hbase12:38117] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-28 04:58:36,252 DEBUG [M:0;jenkins-hbase12:38117] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:58:36,252 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.10 KB heapSize=393.33 KB 2023-04-28 04:58:36,330 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 04:58:36,330 INFO [RS:0;jenkins-hbase12:37663] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase12.apache.org,37663,1682657733313; zookeeper connection closed. 2023-04-28 04:58:36,330 ERROR [Listener at localhost.localdomain/36983-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a412cf6 rejected from java.util.concurrent.ThreadPoolExecutor@3ef1d84e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1785] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 04:58:36,331 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): regionserver:37663-0x100f60986970001, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 04:58:36,331 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@522434cd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@522434cd 2023-04-28 04:58:36,331 ERROR [Listener at localhost.localdomain/36983-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4d7f7d02 rejected from java.util.concurrent.ThreadPoolExecutor@3ef1d84e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1785] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 04:58:36,332 INFO [Listener at localhost.localdomain/36983] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-28 04:58:36,662 INFO [M:0;jenkins-hbase12:38117] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.10 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e050ad5124574d298f415ce858932ee8 2023-04-28 04:58:36,673 INFO [M:0;jenkins-hbase12:38117] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e050ad5124574d298f415ce858932ee8 2023-04-28 04:58:36,674 DEBUG [M:0;jenkins-hbase12:38117] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e050ad5124574d298f415ce858932ee8 as hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e050ad5124574d298f415ce858932ee8 2023-04-28 04:58:36,679 INFO [M:0;jenkins-hbase12:38117] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e050ad5124574d298f415ce858932ee8 2023-04-28 04:58:36,679 INFO [M:0;jenkins-hbase12:38117] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42461/user/jenkins/test-data/ab79c18e-3978-219f-74ab-a8bcb145f56c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e050ad5124574d298f415ce858932ee8, entries=65, sequenceid=572, filesize=24.7 K 2023-04-28 04:58:36,680 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegion(2948): Finished flush of dataSize ~345.10 KB/353382, heapSize ~393.31 KB/402752, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 428ms, sequenceid=572, compaction requested=false 2023-04-28 04:58:36,681 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 04:58:36,681 DEBUG [M:0;jenkins-hbase12:38117] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 04:58:36,685 INFO [M:0;jenkins-hbase12:38117] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-28 04:58:36,685 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-28 04:58:36,686 INFO [M:0;jenkins-hbase12:38117] ipc.NettyRpcServer(158): Stopping server on /136.243.104.168:38117 2023-04-28 04:58:36,693 DEBUG [M:0;jenkins-hbase12:38117] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase12.apache.org,38117,1682657732105 already deleted, retry=false 2023-04-28 04:58:36,885 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 04:58:36,885 INFO [M:0;jenkins-hbase12:38117] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase12.apache.org,38117,1682657732105; zookeeper connection closed. 2023-04-28 04:58:36,886 ERROR [Listener at localhost.localdomain/36983-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@711efb92 rejected from java.util.concurrent.ThreadPoolExecutor@460b196e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2989] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 04:58:36,886 DEBUG [Listener at localhost.localdomain/36983-EventThread] zookeeper.ZKWatcher(600): master:38117-0x100f60986970000, quorum=127.0.0.1:52932, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 04:58:36,886 ERROR [Listener at localhost.localdomain/36983-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@45159462 rejected from java.util.concurrent.ThreadPoolExecutor@460b196e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2989] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 04:58:36,889 WARN [Listener at localhost.localdomain/36983] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-28 04:58:36,894 INFO [Listener at localhost.localdomain/36983] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-28 04:58:37,005 WARN [BP-557455666-136.243.104.168-1682657727979 heartbeating to localhost.localdomain/127.0.0.1:42461] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-28 04:58:37,005 WARN [BP-557455666-136.243.104.168-1682657727979 heartbeating to localhost.localdomain/127.0.0.1:42461] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-557455666-136.243.104.168-1682657727979 (Datanode Uuid 8cc9c9b5-5dd2-430f-8a4b-623291ea4ba7) service to localhost.localdomain/127.0.0.1:42461 2023-04-28 04:58:37,008 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/cluster_ed675f99-8f25-1dee-0ca7-08f146b620e3/dfs/data/data1/current/BP-557455666-136.243.104.168-1682657727979] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-28 04:58:37,008 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e0ac4a0b-eecf-cf58-c1d6-44d2f9103ca6/cluster_ed675f99-8f25-1dee-0ca7-08f146b620e3/dfs/data/data2/current/BP-557455666-136.243.104.168-1682657727979] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-28 04:58:37,039 INFO [Listener at localhost.localdomain/36983] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-28 04:58:37,194 INFO [Listener at localhost.localdomain/36983] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-28 04:58:37,332 INFO [Listener at localhost.localdomain/36983] hbase.HBaseTestingUtility(1293): Minicluster is down